20 Commits

Author SHA1 Message Date
ext.jeremy.guillot@maxicoffee.domains
6065eb0eb1 feat(activity): mises à jour temps réel des jobs via Mercure
- Ajoute jobId dans ChapterScrapingStarted et ChapterScrapingFailed
- Publie job.created (PENDING) depuis ScrapeChapterStateProcessor
- Publie job.status_changed (in_progress/completed/failed) depuis ScrapingEventSubscriber
- Gère job.created et job.status_changed dans activityStore : ajout instantané et suppression différée (1.5s)
2026-03-17 16:19:40 +01:00
ext.jeremy.guillot@maxicoffee.domains
84c4557abf refactor(scraping): job PENDING dès le POST HTTP, handler sans Doctrine
- ScrapingJob: mangaId/chapterNumber/sourceId optionnels (nullable) pour
  permettre la création en PENDING sans lookup DB dans le StateProcessor
- ScrapeChapter: ajoute jobId (pré-généré par le StateProcessor)
- ScrapeChapterStateProcessor: crée et persiste le job PENDING avant
  dispatch; injecte JobRepositoryInterface uniquement
- ScrapeChapterHandler: supprime EntityManagerInterface, beginTransaction/
  commit/rollback; charge le job existant via jobId, complete() sur succès
  seulement, fail() si toutes les sources échouent
- ScrapeChapterHandlerTest: pré-crée le job, passe jobId dans la commande,
  supprime le mock EntityManagerInterface
- ScrapeChapterTest: accès aux messages via static InMemoryMessageBus,
  vérifie la présence du jobId dans la commande dispatchée
2026-03-17 16:19:18 +01:00
ext.jeremy.guillot@maxicoffee.domains
ec4a8be934 feat(system): ajouter filtre par statut dans la page Logs
All checks were successful
Deploy / deploy (push) Successful in 2m30s
- Filtre toolbar : Échecs / Terminés / Tous (défaut : Échecs)
- Badge statut sur chaque LogItem (vert Terminé / rouge Échec)
- deleteAllLogs respecte le filtre actif
2026-03-16 15:00:12 +01:00
8443120c2f Merge pull request 'feat(system): implémenter la page Logs des erreurs de scraping' (#26) from feat/system-logs into main
All checks were successful
Deploy / deploy (push) Successful in 2m32s
Reviewed-on: #26
2026-03-16 14:50:14 +01:00
7a8f749f3f Merge branch 'main' into feat/system-logs 2026-03-16 14:50:04 +01:00
ext.jeremy.guillot@maxicoffee.domains
670e3f5315 feat(system): implémenter la page Logs des erreurs de scraping
- Nouveau domaine `system` avec `logsStore` (Pinia) filtré sur
  status=failed&type=scraping_job, tri, pagination et suppression
- Composant `LogItem` : affiche titre manga, chapitre, date, durée,
  domaine source (lien vers page d'édition), badge type scraping,
  slug utilisé, message d'erreur expandable
- Page `LogsPage` : toolbar avec badge total, dropdown tri, rafraîchir,
  tout supprimer ; charge les ContentSources pour enrichir l'affichage
- Route /system/logs branchée sur LogsPage
- ApiJobRepository : ajout du paramètre `type` dans getJobs
- Job entity : ajout des champs startedAt et completedAt
2026-03-16 14:43:19 +01:00
4398170989 Merge pull request 'feat(setting): implémenter la suppression d'une ContentSource' (#25) from feat/delete-content-source into main
All checks were successful
Deploy / deploy (push) Successful in 2m57s
Reviewed-on: #25
2026-03-16 00:27:59 +01:00
ext.jeremy.guillot@maxicoffee.domains
fc4ab68e8b feat(setting): implémenter la suppression d'une ContentSource
- Ajoute DeleteContentSourceCommand + CommandHandler (CQRS)
- Expose DELETE /api/content-sources/{id} via API Platform (Resource, Provider, Processor)
- Ajoute 2 tests Feature (204 succès, 404 not found)
- Frontend : méthode delete() dans le repository, action deleteSource() dans le store
- Nouveau composant ContentSourceDeleteModal (modale de confirmation)
- Bouton Supprimer dans la toolbar de ScrapperEdit (visible en mode édition uniquement)
2026-03-16 00:27:31 +01:00
36f873aaca Merge pull request 'feat/scrapers-content-sources-healthcheck' (#24) from feat/scrapers-content-sources-healthcheck into main
All checks were successful
Deploy / deploy (push) Successful in 3m17s
Reviewed-on: #24
2026-03-16 00:11:52 +01:00
ext.jeremy.guillot@maxicoffee.domains
874003eb35 fix(scraping): corriger les 403 sur les images avec protection anti-hotlink
- Ajouter le header Referer (origin de l'image) dans ImageDownloader pour les téléchargements backend
- Ajouter referrerpolicy="no-referrer" sur les <img> de la modale de test pour les previews navigateur
2026-03-16 00:11:17 +01:00
ext.jeremy.guillot@maxicoffee.domains
01474c264b feat(scraping): implémenter le health check de tous les scrapers
- Commande CheckAllScrapersHealth + handler avec ports dédiés
- Value Object ContentSourceHealthCheckData
- Resource API Platform et State Processor
- Adapters InMemory et tests unitaires + fonctionnels
2026-03-16 00:11:17 +01:00
ext.jeremy.guillot@maxicoffee.domains
795cbeccc3 feat(setting): étendre ContentSource avec champs de test et domain model
- Ajouter testSlug, testChapterNumber, baseUrl sur ContentSource (entité, domain model, migration)
- Exposer ces champs dans les Resources, Processors, Providers et Mapper
- Mettre à jour store Pinia, repository API et composants Vue (form, card, liste)
2026-03-16 00:11:17 +01:00
b0ce36096f Merge pull request 'feat(ui): harmoniser les pages Scrapers sur le design system Mangarr' (#23) from feat/ui-scrapers-harmonization into main
All checks were successful
Deploy / deploy (push) Successful in 2m58s
Reviewed-on: #23
2026-03-15 22:52:56 +01:00
ext.jeremy.guillot@maxicoffee.domains
da8a19cbcb feat(ui): harmoniser les pages Scrapers sur le design system Mangarr
- Layout canonique px-6 py-8 + sections border-t (suppression container mx-auto)
- Toolbar : label titre + bouton retour (ScrapperEdit) + boutons actions (ScrapperConfigurations)
- Bouton submit déplacé dans la toolbar droite via defineExpose/ref
- ContentSourceForm aplati (suppression du wrapper carte et du header)
- Séparation des sections du formulaire par border-t
- Suppression de tous les rounded-* sur les 4 composants
- Suppression du bloc debug "aucune source" et du h1 volant
2026-03-15 22:52:23 +01:00
ext.jeremy.guillot@maxicoffee.domains
367b361eef fix(manga): afficher la plage de chapitres au lieu du numéro de volume dans la liste
All checks were successful
Deploy / deploy (push) Successful in 2m58s
Pour les chapitres regroupés en volume (isVolumeGroup), la colonne "#" affichait
"Vol. X" au lieu du numéro/plage de chapitres. Remplacé par volumeChaptersRange.
2026-03-15 22:21:19 +01:00
ext.jeremy.guillot@maxicoffee.domains
9c5ae4bf16 fix(scheduler): désactiver MainSchedule legacy au profit de MonitoringSchedule DDD
All checks were successful
Deploy / deploy (push) Successful in 2m53s
MainSchedule (toutes les 6h) et MonitoringSchedule (toutes les 2h) tournaient
en parallèle sur les mêmes mangas surveillés, causant des doubles appels MangaDex
et des doublons de scraping.
2026-03-15 22:08:46 +01:00
ext.jeremy.guillot@maxicoffee.domains
6b58e94fc3 fix(manga): corriger le conflit de shortName sur MangaDiscoverResource
All checks were successful
Deploy / deploy (push) Successful in 2m56s
2026-03-15 21:56:41 +01:00
e78bc890ef Merge pull request 'feat(manga): implémenter la page Découvrir avec recommandations MangaDex' (#22) from feat/discover-page into main
All checks were successful
Deploy / deploy (push) Successful in 2m50s
Reviewed-on: #22
2026-03-15 21:44:43 +01:00
47c33d549b Merge branch 'main' into feat/discover-page 2026-03-15 21:44:28 +01:00
1478b460ba Merge pull request 'style(manga): refondre la page d'ajout de manga sur le design system' (#21) from style/add-manga-ui-redesign into main
All checks were successful
Deploy / deploy (push) Successful in 2m45s
Reviewed-on: #21
2026-03-15 20:56:29 +01:00
60 changed files with 2265 additions and 622 deletions

View File

@@ -1,13 +1,17 @@
import { defineStore } from 'pinia';
import { Job } from '../../domain/entities/job';
import { ApiJobRepository } from '../../infrastructure/api/ApiJobRepository';
const jobRepository = new ApiJobRepository();
const ACTIVE_STATUSES = ['pending', 'in_progress'];
export const useActivityStore = defineStore('activity', {
state: () => ({
jobs: [],
loading: false,
error: null,
mercureEventSource: null,
// Pagination
currentPage: 1,
totalPages: 0,
@@ -15,21 +19,15 @@ export const useActivityStore = defineStore('activity', {
limit: 20,
hasNextPage: false,
hasPreviousPage: false,
// Filtres
filter: {
status: ['pending', 'in_progress'], // Par défaut, ne montrer que les actifs
sortBy: 'createdAt',
sortOrder: 'DESC'
}
// Tri
sortBy: 'createdAt',
sortOrder: 'DESC',
}),
getters: {
activeJobs: state => state.jobs.filter(job => job.isActive()),
completedJobs: state => state.jobs.filter(job => job.isCompleted()),
failedJobs: state => state.jobs.filter(job => job.hasError()),
isLoading: state => state.loading,
hasError: state => !!state.error,
// Getters pour la pagination
paginationInfo: state => ({
currentPage: state.currentPage,
totalPages: state.totalPages,
@@ -41,44 +39,25 @@ export const useActivityStore = defineStore('activity', {
},
actions: {
/**
* Charge la liste des jobs selon les filtres actuels
* @param {number} page - Numéro de page optionnel
*/
async loadJobs(page = null) {
this.loading = true;
this.error = null;
try {
const options = {
const jobCollection = await jobRepository.getJobs({
page: page || this.currentPage,
limit: this.limit,
sortBy: this.filter.sortBy,
sortOrder: this.filter.sortOrder,
status: this.filter.status
};
sortBy: this.sortBy,
sortOrder: this.sortOrder,
status: ACTIVE_STATUSES,
});
const jobCollection = await jobRepository.getJobs(options);
// Mettre à jour les données
this.jobs = jobCollection.items;
this.currentPage = jobCollection.page;
this.total = jobCollection.total;
this.hasNextPage = jobCollection.hasNextPage;
this.hasPreviousPage = jobCollection.hasPreviousPage;
// Calculer le nombre total de pages
this.totalPages = Math.ceil(this.total / this.limit);
console.log('Store updated with:', {
jobs: this.jobs.length,
currentPage: this.currentPage,
total: this.total,
limit: this.limit,
totalPages: this.totalPages,
hasNextPage: this.hasNextPage,
hasPreviousPage: this.hasPreviousPage
});
} catch (error) {
this.error = error.message;
console.error('Error loading jobs:', error);
@@ -87,10 +66,6 @@ export const useActivityStore = defineStore('activity', {
}
},
/**
* Va à une page spécifique
* @param {number} page
*/
async goToPage(page) {
if (page >= 1 && page <= this.totalPages && page !== this.currentPage) {
this.currentPage = page;
@@ -98,39 +73,26 @@ export const useActivityStore = defineStore('activity', {
}
},
/**
* Met à jour les filtres et recharge la liste
* @param {Object} filter
*/
async updateFilter(filter) {
this.filter = { ...this.filter, ...filter };
this.currentPage = 1; // Retourner à la première page lors du changement de filtre
async updateSort(sortBy, sortOrder) {
this.sortBy = sortBy;
this.sortOrder = sortOrder;
this.currentPage = 1;
await this.loadJobs(1);
},
/**
* Met à jour la limite par page
* @param {number} limit
*/
async updateLimit(limit) {
this.limit = limit;
this.currentPage = 1; // Retourner à la première page
this.currentPage = 1;
await this.loadJobs(1);
},
/**
* Supprime un job par son ID
* @param {string} id
*/
async deleteJob(id) {
this.loading = true;
this.error = null;
try {
await jobRepository.deleteJob(id);
// Supprimer le job de la liste locale
this.jobs = this.jobs.filter(job => job.id !== id);
// Recharger la page courante pour avoir les bons totaux
await this.loadJobs(this.currentPage);
} catch (error) {
this.error = error.message;
@@ -140,17 +102,75 @@ export const useActivityStore = defineStore('activity', {
}
},
/**
* Supprime tous les jobs correspondant aux critères
* @param {Object} criteria
*/
updateJobProgress(jobId, progress) {
const job = this.jobs.find(j => j.id === jobId);
if (job) job.progress = progress;
},
handleJobCreated(data) {
const alreadyExists = this.jobs.some(j => j.id === data.id);
if (alreadyExists) return;
const job = Job.create({
id: data.id,
type: data.type_job,
status: data.status,
createdAt: data.createdAt,
context: data.context,
attempts: data.attempts,
maxAttempts: data.maxAttempts,
});
this.jobs.unshift(job);
this.total += 1;
this.totalPages = Math.ceil(this.total / this.limit);
},
handleJobStatusChange(jobId, newStatus) {
const job = this.jobs.find(j => j.id === jobId);
if (!job) return;
if (newStatus === 'in_progress') {
job.status = 'in_progress';
} else {
setTimeout(() => {
this.jobs = this.jobs.filter(j => j.id !== jobId);
this.total = Math.max(0, this.total - 1);
this.totalPages = Math.ceil(this.total / this.limit);
}, 1500);
}
},
subscribeMercure() {
if (this.mercureEventSource) return;
const url = new URL('/.well-known/mercure', window.location.origin);
url.searchParams.append('topic', 'jobs/activity');
this.mercureEventSource = new EventSource(url.toString());
this.mercureEventSource.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.type === 'job.created') {
this.handleJobCreated(data);
} else if (data.type === 'job.progress_updated') {
this.updateJobProgress(data.jobId, data.progress);
} else if (data.type === 'job.status_changed') {
this.handleJobStatusChange(data.jobId, data.status);
}
};
},
unsubscribeMercure() {
if (this.mercureEventSource) {
this.mercureEventSource.close();
this.mercureEventSource = null;
}
},
async deleteJobs(criteria = {}) {
this.loading = true;
this.error = null;
try {
const deleted = await jobRepository.deleteJobs(criteria);
// Recharger la liste après suppression
await this.loadJobs(this.currentPage);
return deleted;
} catch (error) {
@@ -160,26 +180,5 @@ export const useActivityStore = defineStore('activity', {
this.loading = false;
}
},
/**
* Supprime tous les jobs terminés
*/
async deleteCompletedJobs() {
return this.deleteJobs({ status: ['COMPLETED'] });
},
/**
* Supprime tous les jobs en erreur
*/
async deleteFailedJobs() {
return this.deleteJobs({ status: ['ERROR'] });
},
/**
* Supprime tous les jobs
*/
async deleteAllJobs() {
return this.deleteJobs({});
}
}
});

View File

@@ -10,6 +10,8 @@ export class Job {
failureReason = null,
createdAt = new Date().toISOString(),
updatedAt = new Date().toISOString(),
startedAt = null,
completedAt = null,
attempts = 0,
maxAttempts = 1,
context = {}
@@ -23,6 +25,8 @@ export class Job {
this.error = failureReason ?? error;
this.createdAt = createdAt;
this.updatedAt = updatedAt;
this.startedAt = startedAt;
this.completedAt = completedAt;
this.attempts = attempts;
this.maxAttempts = maxAttempts;
this.context = context;

View File

@@ -13,7 +13,7 @@ export class ApiJobRepository extends JobRepositoryInterface {
* @returns {Promise<JobCollection>} Collection de jobs
*/
async getJobs(options = {}) {
const { page = 1, limit = 100, sortBy = 'createdAt', sortOrder = 'DESC', status = [] } = options;
const { page = 1, limit = 100, sortBy = 'createdAt', sortOrder = 'DESC', status = [], type = null } = options;
try {
let url = `/api/jobs?page=${page}&limit=${limit}&sortBy=${sortBy}&sortOrder=${sortOrder}`;
@@ -23,6 +23,11 @@ export class ApiJobRepository extends JobRepositoryInterface {
url += `&status=${status.join(',')}`;
}
// Ajouter le filtre de type si fourni
if (type) {
url += `&type=${type}`;
}
const response = await fetch(url);
if (!response.ok) {

View File

@@ -1,169 +1,153 @@
<template>
<div class="overflow-y-auto h-full">
<Toolbar :config="toolbarConfig" class="mb-6" />
<div class="flex flex-col h-full">
<Toolbar :config="toolbarConfig" />
<div v-if="activityStore.loading" class="flex justify-center py-8">
<div class="animate-spin rounded-full h-10 w-10 border-t-2 border-b-2 border-indigo-500"></div>
</div>
<div class="overflow-y-auto flex-1">
<!-- Loading -->
<div v-if="loading" class="flex justify-center py-12">
<div class="animate-spin h-10 w-10 border-b-2 border-indigo-500 rounded-full"></div>
</div>
<div v-else-if="activityStore.error" class="bg-red-100 dark:bg-red-900/20 border-l-4 border-red-500 text-red-700 dark:text-red-400 p-4 mb-6">
<p>{{ activityStore.error }}</p>
</div>
<!-- Error -->
<div v-else-if="activityStore.error" class="px-6 py-8">
<div class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-4">
<p class="text-red-800 dark:text-red-200">{{ activityStore.error }}</p>
</div>
</div>
<div v-else class="container mx-auto p-2">
<div class="bg-white dark:bg-gray-800 overflow-hidden shadow rounded-lg">
<div class="overflow-x-auto">
<table class="min-w-full bg-white dark:bg-gray-800">
<!-- Content -->
<section v-else class="border-t border-gray-200 dark:border-gray-700">
<!-- Empty -->
<div v-if="activityStore.jobs.length === 0" class="flex flex-col items-center justify-center py-20 text-gray-400 dark:text-gray-500">
<ClockIcon class="w-12 h-12 mb-3" />
<p class="text-base">Aucun job en cours ou en attente.</p>
</div>
<!-- Table -->
<div v-else class="overflow-x-auto">
<table class="min-w-full">
<thead>
<tr class="bg-gray-200 dark:bg-gray-700 text-gray-800 dark:text-gray-200">
<th class="w-1/12 py-3 px-4 text-left">
<input
type="checkbox"
class="form-checkbox h-5 w-5 text-green-600"
@change="toggleSelectAll" />
</th>
<th class="w-2/12 py-3 px-4 text-left">Type</th>
<th class="w-2/12 py-3 px-4 text-left">Statut</th>
<th class="w-3/12 py-3 px-4 text-left">Informations</th>
<th class="w-3/12 py-3 px-4 text-left">Progression</th>
<th class="w-1/12 py-3 px-4 text-left">Actions</th>
<tr class="border-b border-gray-200 dark:border-gray-700 text-xs font-semibold text-gray-400 dark:text-gray-500 uppercase tracking-wider">
<th class="w-2/11 py-3 px-6 text-left">Type</th>
<th class="w-2/11 py-3 px-4 text-left">Statut</th>
<th class="w-3/11 py-3 px-4 text-left">Informations</th>
<th class="w-3/11 py-3 px-4 text-left">Progression</th>
<th class="w-1/11 py-3 px-4 text-left">Actions</th>
</tr>
</thead>
<tbody class="text-gray-700 dark:text-gray-300">
<template v-if="activityStore.jobs.length === 0">
<tr>
<td colspan="6" class="py-8 px-4 text-center text-gray-500">
<div class="flex flex-col items-center">
<ClockIcon class="h-12 w-12 text-gray-300 dark:text-gray-600 mb-4" />
<p class="text-lg font-medium dark:text-gray-300">Aucune activité trouvée</p>
<p class="text-sm dark:text-gray-400">Aucune activité ne correspond aux filtres actuels.</p>
</div>
</td>
</tr>
</template>
<template v-else>
<JobItem
v-for="job in activityStore.jobs"
:key="job.id"
:job="job"
@delete="deleteJob" />
</template>
<tbody class="divide-y divide-gray-100 dark:divide-gray-700/50 text-gray-700 dark:text-gray-300">
<JobItem
v-for="job in activityStore.jobs"
:key="job.id"
:job="job"
@delete="deleteJob" />
</tbody>
</table>
</div>
<!-- Pagination -->
<Pagination
v-if="activityStore.total > activityStore.limit"
v-if="total > activityStore.limit"
:current-page="activityStore.currentPage"
:total-pages="activityStore.totalPages"
:total="activityStore.total"
:total="total"
:limit="activityStore.limit"
:has-next-page="activityStore.hasNextPage"
:has-previous-page="activityStore.hasPreviousPage"
@page-change="changePage" />
</div>
</section>
</div>
</div>
</template>
<script setup>
import { ArrowPathIcon, ClockIcon, FunnelIcon, TrashIcon } from '@heroicons/vue/24/outline';
import { computed, onMounted, ref } from 'vue';
import { ArrowPathIcon, BarsArrowDownIcon, ClockIcon, TrashIcon } from '@heroicons/vue/24/outline';
import { storeToRefs } from 'pinia';
import { computed, onMounted, onUnmounted } from 'vue';
import Pagination from '../../../../shared/components/ui/Pagination.vue';
import Toolbar from '../../../../shared/components/ui/Toolbar.vue';
import { useActivityStore } from '../../application/store/activityStore';
import JobItem from '../components/JobItem.vue';
const activityStore = useActivityStore();
const selectedAll = ref(false);
const activityStore = useActivityStore();
// Statuts disponibles pour le filtre
const statusOptions = [
{ value: ['pending', 'in_progress'], label: 'Actifs' },
{ value: ['pending', 'in_progress', 'completed', 'failed'], label: 'Tous' },
{ value: ['completed'], label: 'Terminés' },
{ value: ['failed'], label: 'En erreur' },
{ value: ['pending'], label: 'En attente' },
{ value: ['in_progress'], label: 'En cours' }
];
const { sortBy, sortOrder, total, loading } = storeToRefs(activityStore);
// Index du statut actif (par défaut "Actifs")
const activeStatusIndex = ref(0);
const isSortSelected = (by, order) => sortBy.value === by && sortOrder.value === order;
// Configuration de la toolbar réactive
const toolbarConfig = computed(() => ({
leftSection: [
{
icon: FunnelIcon,
type: 'dropdown',
label: statusOptions[activeStatusIndex.value].label,
active: false,
items: statusOptions.map((option, index) => ({
label: option.label,
isSelected: index === activeStatusIndex.value,
onClick: () => setStatusFilter(index)
}))
}
],
rightSection: [
{
icon: ArrowPathIcon,
type: 'button',
label: 'Rafraîchir',
onClick: refreshJobs
},
{
icon: TrashIcon,
type: 'button',
label: 'Supprimer visibles',
onClick: deleteVisibleJobs
}
]
}));
const toolbarConfig = computed(() => ({
leftSection: [
{ type: 'label', text: 'Activité', class: 'text-sm font-medium' },
{ type: 'label', text: `(${total.value})`, class: 'text-sm text-gray-400' },
],
rightSection: [
{
type: 'dropdown',
icon: BarsArrowDownIcon,
label: 'Trier',
items: [
{
label: 'Plus récent',
isSelected: isSortSelected('createdAt', 'DESC'),
onClick: () => activityStore.updateSort('createdAt', 'DESC'),
},
{
label: 'Plus ancien',
isSelected: isSortSelected('createdAt', 'ASC'),
onClick: () => activityStore.updateSort('createdAt', 'ASC'),
},
{
label: 'Par type',
isSelected: isSortSelected('type', 'ASC'),
onClick: () => activityStore.updateSort('type', 'ASC'),
},
{
label: 'Par statut',
isSelected: isSortSelected('status', 'ASC'),
onClick: () => activityStore.updateSort('status', 'ASC'),
},
],
},
{
type: 'button',
icon: ArrowPathIcon,
label: 'Rafraîchir',
disabled: loading.value,
onClick: () => activityStore.loadJobs(),
},
{
type: 'button',
icon: TrashIcon,
label: 'Supprimer visibles',
disabled: loading.value || total.value === 0,
onClick: deleteVisibleJobs,
},
],
}));
onMounted(() => {
loadJobs();
});
onMounted(() => {
activityStore.loadJobs();
activityStore.subscribeMercure();
});
function loadJobs() {
activityStore.loadJobs();
onUnmounted(() => {
activityStore.unsubscribeMercure();
});
function changePage(page) {
activityStore.goToPage(page);
}
function deleteJob(id) {
if (confirm('Voulez-vous vraiment supprimer ce job ?')) {
activityStore.deleteJob(id);
}
}
function refreshJobs() {
loadJobs();
}
function changePage(page) {
activityStore.goToPage(page);
}
function toggleSelectAll() {
selectedAll.value = !selectedAll.value;
// La logique pour sélectionner tous les jobs serait ajoutée ici
}
function setStatusFilter(index) {
if (index >= 0 && index < statusOptions.length) {
activeStatusIndex.value = index;
activityStore.updateFilter({ status: statusOptions[index].value });
}
}
function deleteJob(id) {
if (confirm('Voulez-vous vraiment supprimer ce job ?')) {
activityStore.deleteJob(id);
}
}
function deleteVisibleJobs() {
if (activityStore.jobs.length === 0) {
return;
}
const statusLabel = statusOptions[activeStatusIndex.value].label.toLowerCase();
if (confirm(`Voulez-vous vraiment supprimer tous les jobs ${statusLabel} ?`)) {
activityStore.deleteJobs({ status: activityStore.filter.status });
}
function deleteVisibleJobs() {
if (activityStore.jobs.length === 0) return;
if (confirm('Voulez-vous vraiment supprimer tous les jobs visibles ?')) {
activityStore.deleteJobs({ status: ['pending', 'in_progress'] });
}
}
</script>

View File

@@ -1,7 +1,7 @@
<template>
<tr class="border-t dark:border-gray-700 hover:bg-green-100 dark:hover:bg-green-900/20">
<td class="px-4 py-2 text-gray-900 dark:text-gray-100" :class="{ 'text-green-500 dark:text-green-400': chapter.isAvailable }">
<template v-if="chapter.isVolumeGroup">Vol. {{ chapter.volume }}</template>
<template v-if="chapter.isVolumeGroup">{{ chapter.volumeChaptersRange }}</template>
<template v-else>{{ String(chapter.number).padStart(2, '0') }}</template>
</td>
<td class="px-4 py-2 w-full text-left text-gray-900 dark:text-gray-100">

View File

@@ -23,7 +23,15 @@ export const useContentSourceStore = defineStore('contentSource', {
importing: false,
exporting: false,
importError: null,
exportError: null
exportError: null,
// Health check state
checkingHealth: false,
checkHealthError: null,
// Delete state
deleting: false,
deleteError: null,
}),
getters: {
@@ -168,12 +176,64 @@ export const useContentSourceStore = defineStore('contentSource', {
}
},
// Delete a source
async deleteSource(id) {
if (this.deleting) return;
this.deleting = true;
this.deleteError = null;
try {
await contentSourceRepository.delete(id);
this.sources = this.sources.filter(source => source.id !== id);
if (this.currentSource && this.currentSource.id === id) {
this.currentSource = null;
}
} catch (error) {
this.deleteError = error.message;
console.error('Erreur lors de la suppression de la source:', error);
throw error;
} finally {
this.deleting = false;
}
},
// Clear current source
clearCurrentSource() {
this.currentSource = null;
this.currentSourceError = null;
},
// Check all scrapers health
async checkAllHealth() {
if (this.checkingHealth) return;
this.checkingHealth = true;
this.checkHealthError = null;
try {
await contentSourceRepository.checkAllHealth();
} catch (error) {
this.checkHealthError = error.message;
console.error('Erreur lors du health check:', error);
throw error;
} finally {
this.checkingHealth = false;
}
},
// Update health status of a single source (called from Mercure)
updateSourceHealth(sourceId, status, error = null) {
const index = this.sources.findIndex(s => s.id === sourceId);
if (index !== -1) {
this.sources[index] = {
...this.sources[index],
healthStatus: status,
healthLastError: error,
};
}
},
// Clear errors
clearErrors() {
this.sourcesError = null;
@@ -181,6 +241,7 @@ export const useContentSourceStore = defineStore('contentSource', {
this.saveError = null;
this.importError = null;
this.exportError = null;
this.checkHealthError = null;
}
}
});

View File

@@ -0,0 +1,6 @@
export const ScraperHealthStatus = {
UNKNOWN: 'unknown',
OK: 'ok',
KO: 'ko',
TESTING: 'testing',
};

View File

@@ -82,6 +82,28 @@ export class ApiContentSourceRepository {
}
}
/**
* Déclenche le test de santé de tous les scrapers
*/
async checkAllHealth() {
try {
await this.apiClient.post('/scraping/check-all-health', {});
} catch (error) {
throw new Error(error.response?.data?.message || 'Erreur lors du lancement du health check');
}
}
/**
* Supprime une source de contenu
*/
async delete(id) {
try {
await this.apiClient.delete(`/content-sources/${id}`);
} catch (error) {
throw new Error(error.response?.data?.message || 'Erreur lors de la suppression de la source');
}
}
/**
* Teste une configuration de scraper
*/

View File

@@ -1,7 +1,7 @@
<template>
<div
@click="$emit('edit', source)"
class="bg-white dark:bg-gray-800 rounded-lg shadow-md border border-gray-200 dark:border-gray-700 p-6 hover:shadow-lg transition-shadow duration-200 cursor-pointer">
class="bg-white dark:bg-gray-800 shadow-md border border-gray-200 dark:border-gray-700 p-6 hover:shadow-lg transition-shadow duration-200 cursor-pointer">
<!-- Header avec URL et icône externe -->
<div class="flex items-center justify-between mb-4">
<h3 class="text-lg font-semibold text-gray-900 dark:text-white truncate" :title="source.cleanBaseUrl">
@@ -20,16 +20,24 @@
<!-- Badge type de scraping -->
<span
:class="getScrapingTypeBadgeClass(source.scrapingType)"
class="px-2 py-1 text-xs font-medium rounded-md">
class="px-2 py-1 text-xs font-medium">
{{ source.scrapingType?.toLowerCase() || 'N/A' }}
</span>
<!-- Badge orientation basé sur les sélecteurs -->
<span
:class="getOrientationBadgeClass(source)"
class="px-2 py-1 text-xs font-medium rounded-md">
class="px-2 py-1 text-xs font-medium">
{{ getOrientation(source) }}
</span>
<!-- Badge health status -->
<span
:class="getHealthBadgeClass(source.healthStatus)"
class="px-2 py-1 text-xs font-medium"
:title="source.healthLastError || ''">
{{ getHealthLabel(source.healthStatus) }}
</span>
</div>
@@ -39,6 +47,7 @@
<script setup>
import { ArrowTopRightOnSquareIcon } from '@heroicons/vue/24/outline';
import { ScraperHealthStatus } from '../../domain/model/ScraperHealthStatus';
defineProps({
source: {
@@ -86,4 +95,26 @@ const getOrientationBadgeClass = (source) => {
return 'bg-gray-100 text-gray-800 dark:bg-gray-700 dark:text-gray-300';
}
};
const getHealthLabel = (status) => {
switch (status) {
case ScraperHealthStatus.OK: return '✓ ok';
case ScraperHealthStatus.KO: return '✗ ko';
case ScraperHealthStatus.TESTING: return '⟳ test';
default: return '? unknown';
}
};
const getHealthBadgeClass = (status) => {
switch (status) {
case ScraperHealthStatus.OK:
return 'bg-green-100 text-green-800 dark:bg-green-900 dark:text-green-300';
case ScraperHealthStatus.KO:
return 'bg-red-100 text-red-800 dark:bg-red-900 dark:text-red-300';
case ScraperHealthStatus.TESTING:
return 'bg-yellow-100 text-yellow-800 dark:bg-yellow-900 dark:text-yellow-300';
default:
return 'bg-gray-100 text-gray-600 dark:bg-gray-700 dark:text-gray-400';
}
};
</script>

View File

@@ -0,0 +1,123 @@
<template>
<TransitionRoot as="template" :show="isOpen">
<Dialog as="div" class="relative z-50" @close="closeModal">
<TransitionChild
as="template"
enter="ease-out duration-300"
enter-from="opacity-0"
enter-to="opacity-100"
leave="ease-in duration-200"
leave-from="opacity-100"
leave-to="opacity-0"
>
<div class="fixed inset-0 bg-gray-500 dark:bg-gray-900 bg-opacity-75 dark:bg-opacity-80 transition-opacity" />
</TransitionChild>
<div class="fixed inset-0 z-10 overflow-y-auto">
<div class="flex min-h-full items-end justify-center p-4 text-center sm:items-center sm:p-0">
<TransitionChild
as="template"
enter="ease-out duration-300"
enter-from="opacity-0 translate-y-4 sm:translate-y-0 sm:scale-95"
enter-to="opacity-100 translate-y-0 sm:scale-100"
leave="ease-in duration-200"
leave-from="opacity-100 translate-y-0 sm:scale-100"
leave-to="opacity-0 translate-y-4 sm:translate-y-0 sm:scale-95"
>
<DialogPanel class="relative transform overflow-hidden rounded-lg bg-white dark:bg-gray-800 px-6 pb-6 pt-6 text-left shadow-xl transition-all sm:my-8 sm:w-full sm:max-w-lg">
<div class="mb-6">
<DialogTitle as="h3" class="text-lg font-semibold leading-6 text-gray-900 dark:text-gray-100">
Supprimer la source de contenu
</DialogTitle>
</div>
<!-- Error state -->
<div v-if="error" class="mb-6 bg-red-100 dark:bg-red-900/20 border border-red-400 dark:border-red-700 text-red-700 dark:text-red-400 px-4 py-3 rounded">
{{ error }}
</div>
<!-- Warning message -->
<div class="mb-6">
<div class="flex items-center mb-4">
<ExclamationTriangleIcon class="h-6 w-6 text-red-500 mr-3" />
<span class="text-sm font-medium text-gray-900 dark:text-gray-100">Action irréversible</span>
</div>
<p class="text-sm text-gray-600 dark:text-gray-400 mb-4">
Êtes-vous sûr de vouloir supprimer la source <strong>{{ source?.baseUrl }}</strong> ?
</p>
<div class="bg-yellow-50 dark:bg-yellow-900/20 border border-yellow-200 dark:border-yellow-700 rounded-md p-4">
<div class="flex">
<ExclamationTriangleIcon class="h-5 w-5 text-yellow-400" />
<div class="ml-3">
<h3 class="text-sm font-medium text-yellow-800 dark:text-yellow-300">
Attention
</h3>
<div class="mt-2 text-sm text-yellow-700 dark:text-yellow-400">
<p>Cette source ne pourra plus être utilisée pour le scraping des chapitres.</p>
</div>
</div>
</div>
</div>
</div>
<!-- Action buttons -->
<div class="mt-6 flex justify-end space-x-3">
<button
type="button"
class="inline-flex justify-center rounded-md border border-gray-300 dark:border-gray-600 bg-white dark:bg-gray-700 px-4 py-2 text-sm font-medium text-gray-700 dark:text-gray-300 shadow-sm hover:bg-gray-50 dark:hover:bg-gray-600 focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2"
@click="closeModal"
:disabled="isLoading"
>
Annuler
</button>
<button
type="button"
class="inline-flex justify-center rounded-md border border-transparent bg-red-600 px-4 py-2 text-sm font-medium text-white shadow-sm hover:bg-red-700 focus:outline-none focus:ring-2 focus:ring-red-500 focus:ring-offset-2 disabled:opacity-50 disabled:cursor-not-allowed"
@click="confirmDelete"
:disabled="isLoading"
>
<ArrowPathIcon v-if="isLoading" class="h-4 w-4 mr-2 animate-spin" />
{{ isLoading ? 'Suppression...' : 'Supprimer définitivement' }}
</button>
</div>
</DialogPanel>
</TransitionChild>
</div>
</div>
</Dialog>
</TransitionRoot>
</template>
<script setup>
import { Dialog, DialogPanel, DialogTitle, TransitionChild, TransitionRoot } from '@headlessui/vue';
import { ArrowPathIcon, ExclamationTriangleIcon } from '@heroicons/vue/24/outline';
const props = defineProps({
isOpen: {
type: Boolean,
required: true
},
source: {
type: Object,
default: null
},
isLoading: {
type: Boolean,
default: false
},
error: {
type: String,
default: null
}
});
const emit = defineEmits(['close', 'confirm']);
const closeModal = () => {
emit('close');
};
const confirmDelete = () => {
emit('confirm');
};
</script>

View File

@@ -1,17 +1,7 @@
<template>
<div class="bg-white dark:bg-gray-800 rounded-lg shadow-lg border border-gray-200 dark:border-gray-700">
<!-- Header -->
<div class="bg-gray-50 dark:bg-gray-700 px-6 py-4 border-b border-gray-200 dark:border-gray-600 rounded-t-lg">
<div class="flex items-center space-x-2">
<Cog6ToothIcon class="w-5 h-5 text-gray-600 dark:text-gray-400" />
<h2 class="text-lg font-semibold text-gray-900 dark:text-white">
{{ isEditing ? 'Edit Scrapper Configuration' : 'New Scrapper Configuration' }}
</h2>
</div>
</div>
<div>
<!-- Form -->
<form @submit.prevent="handleSubmit" class="p-6 space-y-6">
<form @submit.prevent="handleSubmit" class="space-y-6">
<!-- Base URL -->
<div>
<label for="baseUrl" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
@@ -22,25 +12,12 @@
v-model="form.baseUrl"
type="url"
required
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="https://example.com" />
</div>
<!-- Image Selector -->
<div>
<label for="imageSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Image Selector
</label>
<input
id="imageSelector"
v-model="form.imageSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".reading-content .page-break img" />
</div>
<!-- Chapter URL Format -->
<div>
<div class="border-t border-gray-200 dark:border-gray-700 pt-6">
<label for="chapterUrlFormat" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Chapter URL Format <span class="text-gray-500">({slug}, {chapterNumber})</span>
</label>
@@ -49,132 +26,132 @@
v-model="form.chapterUrlFormat"
type="text"
required
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="https://example.com/manga/{slug}-{chapterNumber}/" />
</div>
<!-- Next Page Selector -->
<div>
<label for="nextPageSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Next Page Selector <span class="text-gray-500">(let empty if vertical reader)</span>
</label>
<input
id="nextPageSelector"
v-model="form.nextPageSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".next-page" />
<!-- Selectors -->
<div class="border-t border-gray-200 dark:border-gray-700 pt-6 space-y-4">
<div>
<label for="imageSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Image Selector
</label>
<input
id="imageSelector"
v-model="form.imageSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".reading-content .page-break img" />
</div>
<div>
<label for="nextPageSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Next Page Selector <span class="text-gray-500">(laisser vide si lecteur vertical)</span>
</label>
<input
id="nextPageSelector"
v-model="form.nextPageSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".next-page" />
</div>
<div>
<label for="chapterSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Chapter Selector <span class="text-gray-500">(requis pour le scraping Javascript)</span>
</label>
<input
id="chapterSelector"
v-model="form.chapterSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".chapter-selector" />
</div>
</div>
<!-- Chapter Selector -->
<div>
<label for="chapterSelector" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Chapter Selector <span class="text-gray-500">(required for Javascript scraping)</span>
</label>
<input
id="chapterSelector"
v-model="form.chapterSelector"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder=".chapter-selector" />
</div>
<!-- Scraping Type + Token -->
<div class="border-t border-gray-200 dark:border-gray-700 pt-6 space-y-4">
<div>
<label for="scrapingType" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Scraping Type
</label>
<select
id="scrapingType"
v-model="form.scrapingType"
required
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white">
<option value="html">HTML</option>
<option value="javascript">Javascript</option>
</select>
</div>
<!-- Scraping Type -->
<div>
<label for="scrapingType" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Scraping Type
</label>
<select
id="scrapingType"
v-model="form.scrapingType"
required
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white">
<option value="html">HTML</option>
<option value="javascript">Javascript</option>
</select>
</div>
<!-- Token (optionnel) -->
<div>
<label for="token" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Token
</label>
<input
id="token"
v-model="form.token"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="Optional authentication token" />
</div>
<!-- Submit Button -->
<div class="flex justify-end">
<button
type="submit"
:disabled="saving"
class="px-6 py-2 bg-green-600 hover:bg-green-700 disabled:bg-green-400 text-white font-medium rounded-md transition-colors duration-200 flex items-center space-x-2">
<ArrowPathIcon v-if="saving" class="w-4 h-4 animate-spin" />
<span>{{ isEditing ? 'Update Configuration' : 'Create Configuration' }}</span>
<PencilSquareIcon v-if="!saving" class="w-4 h-4" />
</button>
<div>
<label for="token" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Token
</label>
<input
id="token"
v-model="form.token"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="Optional authentication token" />
</div>
</div>
<!-- Error message -->
<div v-if="error" class="text-red-600 dark:text-red-400 text-sm">
<div v-if="error" class="border-t border-gray-200 dark:border-gray-700 pt-6 text-red-600 dark:text-red-400 text-sm">
{{ error }}
</div>
</form>
<!-- Test Configuration Section -->
<div class="border-t border-gray-200 dark:border-gray-600 p-6 bg-gray-50 dark:bg-gray-700 rounded-b-lg">
<div class="flex items-center space-x-2 mb-4">
<div class="border-t border-gray-200 dark:border-gray-700 pt-6 mt-6">
<div class="flex items-center space-x-2 mb-6">
<WrenchScrewdriverIcon class="w-5 h-5 text-gray-600 dark:text-gray-400" />
<h3 class="text-lg font-medium text-gray-900 dark:text-white">Test Configuration</h3>
<h3 class="text-sm font-medium text-gray-900 dark:text-white">Configuration de test (health check)</h3>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4 mb-4">
<div>
<label for="testMangaSlug" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Manga Slug
<label for="testSlug" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Manga Slug <span class="text-gray-500">(enregistré)</span>
</label>
<input
id="testMangaSlug"
v-model="testData.mangaSlug"
id="testSlug"
v-model="form.testSlug"
type="text"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="manga-slug" />
</div>
<div>
<label for="testChapterNumber" class="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Chapter Number
Numéro de chapitre <span class="text-gray-500">(enregistré)</span>
</label>
<input
id="testChapterNumber"
v-model="testData.chapterNumber"
v-model="form.testChapterNumber"
type="number"
step="0.1"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
class="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 focus:ring-blue-500 focus:border-blue-500 dark:bg-gray-700 dark:text-white"
placeholder="1" />
</div>
</div>
<!-- Preview de l'URL qui sera testée -->
<div v-if="generatedTestUrl" class="mb-4 p-3 bg-blue-50 dark:bg-blue-900 border border-blue-200 dark:border-blue-700 rounded-md">
<div class="text-sm text-blue-800 dark:text-blue-200">
<strong>URL qui sera testée :</strong>
<div class="mt-1 font-mono text-xs break-all">{{ generatedTestUrl }}</div>
</div>
<!-- Preview URL -->
<div v-if="generatedTestUrl" class="mb-4 border-t border-gray-200 dark:border-gray-700 pt-4">
<p class="text-xs text-gray-500 dark:text-gray-400 mb-1">URL qui sera testée</p>
<code class="text-xs text-gray-700 dark:text-gray-300 break-all">{{ generatedTestUrl }}</code>
</div>
<button
type="button"
@click="testConfiguration"
:disabled="testing || !canTest"
class="w-full px-4 py-2 bg-blue-600 hover:bg-blue-700 disabled:bg-blue-400 text-white font-medium rounded-md transition-colors duration-200 flex items-center justify-center space-x-2">
class="w-full px-4 py-2 bg-blue-600 hover:bg-blue-700 disabled:bg-blue-400 text-white font-medium transition-colors duration-200 flex items-center justify-center space-x-2">
<ArrowPathIcon v-if="testing" class="w-4 h-4 animate-spin" />
<PlayIcon v-else class="w-4 h-4" />
<span>Test Configuration</span>
<span>Tester maintenant</span>
</button>
</div>
</div>
@@ -183,8 +160,6 @@
<script setup>
import {
ArrowPathIcon,
Cog6ToothIcon,
PencilSquareIcon,
PlayIcon,
WrenchScrewdriverIcon
} from '@heroicons/vue/24/outline';
@@ -216,12 +191,9 @@ const form = ref({
nextPageSelector: '',
chapterSelector: '',
scrapingType: 'html',
token: ''
});
const testData = ref({
mangaSlug: '',
chapterNumber: ''
token: '',
testSlug: '',
testChapterNumber: '',
});
const testing = ref(false);
@@ -229,20 +201,19 @@ const testing = ref(false);
const canTest = computed(() => {
return form.value.baseUrl &&
form.value.chapterUrlFormat &&
testData.value.mangaSlug &&
testData.value.chapterNumber;
form.value.testSlug &&
form.value.testChapterNumber;
});
const generatedTestUrl = computed(() => {
if (!form.value.chapterUrlFormat || !testData.value.mangaSlug || !testData.value.chapterNumber) {
if (!form.value.chapterUrlFormat || !form.value.testSlug || !form.value.testChapterNumber) {
return '';
}
return form.value.chapterUrlFormat
.replace('{slug}', testData.value.mangaSlug)
.replace('{chapterNumber}', testData.value.chapterNumber);
.replace('{slug}', form.value.testSlug)
.replace('{chapterNumber}', form.value.testChapterNumber);
});
// Initialize form with source data if editing, clear if creating new
watch(() => props.source, (newSource) => {
if (newSource) {
form.value = {
@@ -252,10 +223,11 @@ watch(() => props.source, (newSource) => {
nextPageSelector: newSource.nextPageSelector || '',
chapterSelector: newSource.chapterSelector || '',
scrapingType: (newSource.scrapingType || 'html').toLowerCase(),
token: newSource.token || ''
token: newSource.token || '',
testSlug: newSource.testSlug || '',
testChapterNumber: newSource.testChapterNumber ?? '',
};
} else {
// Reset form when no source (creating new)
form.value = {
baseUrl: '',
imageSelector: '',
@@ -263,7 +235,9 @@ watch(() => props.source, (newSource) => {
nextPageSelector: '',
chapterSelector: '',
scrapingType: 'html',
token: ''
token: '',
testSlug: '',
testChapterNumber: '',
};
}
}, { immediate: true });
@@ -272,14 +246,17 @@ const handleSubmit = () => {
emit('submit', { ...form.value });
};
defineExpose({ submitForm: handleSubmit });
const testConfiguration = async () => {
testing.value = true;
try {
await emit('test', {
configuration: { ...form.value },
testData: {
...testData.value,
testUrl: generatedTestUrl.value
mangaSlug: form.value.testSlug,
chapterNumber: form.value.testChapterNumber,
testUrl: generatedTestUrl.value,
}
});
} finally {

View File

@@ -3,72 +3,54 @@
<Toolbar :config="toolbarConfig" />
<div class="overflow-y-auto flex-1">
<div class="container mx-auto px-4 py-6">
<!-- Header -->
<div class="mb-8">
<h1 class="text-3xl font-bold text-gray-900 dark:text-white mb-2">
Scrapper Configurations
</h1>
<p class="text-gray-600 dark:text-gray-400">
Gérez les configurations de scraping pour les différentes sources de manga
</p>
</div>
<div class="px-6 py-8">
<!-- Loading State -->
<div v-if="loadingSources" class="flex justify-center py-12">
<div class="animate-spin rounded-full h-12 w-12 border-b-2 border-blue-500"></div>
<div class="animate-spin h-12 w-12 border-b-2 border-blue-500"></div>
</div>
<!-- Error State -->
<div v-else-if="sourcesError" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 rounded-lg p-4 mb-6">
<div v-else-if="sourcesError" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-4 mb-6">
<div class="flex items-center">
<ExclamationTriangleIcon class="w-5 h-5 text-red-400 mr-2" />
<p class="text-red-800 dark:text-red-200">{{ sourcesError }}</p>
</div>
<button
@click="contentSourceStore.loadSources()"
class="mt-3 px-4 py-2 bg-red-600 text-white rounded hover:bg-red-700">
class="mt-3 px-4 py-2 bg-red-600 text-white hover:bg-red-700">
Réessayer
</button>
</div>
<!-- Debug Info (temporary) -->
<div v-if="!loadingSources && !sourcesError && sources.length === 0" class="bg-blue-50 dark:bg-blue-900 border border-blue-200 dark:border-blue-700 rounded-lg p-4 mb-6">
<p class="text-blue-800 dark:text-blue-200">Aucune source trouvée. Rechargement en cours...</p>
<button
@click="contentSourceStore.loadSources()"
class="mt-2 px-4 py-2 bg-blue-600 text-white rounded hover:bg-blue-700">
Actualiser
</button>
</div>
<!-- Sources Grid -->
<div v-else class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
<!-- Existing Sources -->
<ContentSourceCard
v-for="source in sources"
:key="source.id"
:source="source"
@edit="editSource"
@open-link="openSourceLink" />
<section v-else class="border-t border-gray-200 dark:border-gray-700 pt-6">
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
<!-- Existing Sources -->
<ContentSourceCard
v-for="source in sources"
:key="source.id"
:source="source"
@edit="editSource"
@open-link="openSourceLink" />
<!-- Add New Configuration Card -->
<div
@click="addNewSource"
class="bg-gray-50 dark:bg-gray-700 border-2 border-dashed border-gray-300 dark:border-gray-600 rounded-lg p-6 hover:border-gray-400 dark:hover:border-gray-500 transition-colors cursor-pointer flex flex-col items-center justify-center h-full">
<PlusIcon class="w-8 h-8 text-gray-400 dark:text-gray-500 mb-3" />
<span class="text-lg font-medium text-gray-600 dark:text-gray-400 mb-2">
Add New Configuration
</span>
<!-- Add New Configuration Card -->
<div
@click="addNewSource"
class="bg-gray-50 dark:bg-gray-700 border-2 border-dashed border-gray-300 dark:border-gray-600 p-6 hover:border-gray-400 dark:hover:border-gray-500 transition-colors cursor-pointer flex flex-col items-center justify-center h-full">
<PlusIcon class="w-8 h-8 text-gray-400 dark:text-gray-500 mb-3" />
<span class="text-lg font-medium text-gray-600 dark:text-gray-400 mb-2">
Add New Configuration
</span>
</div>
</div>
</div>
</section>
<!-- Import/Export Success Messages -->
<div v-if="showImportSuccess" class="fixed bottom-4 right-4 bg-green-500 text-white px-4 py-2 rounded-lg shadow-lg">
<div v-if="showImportSuccess" class="fixed bottom-4 right-4 bg-green-500 text-white px-4 py-2 shadow-lg">
Configuration importée avec succès !
</div>
<div v-if="showExportSuccess" class="fixed bottom-4 right-4 bg-blue-500 text-white px-4 py-2 rounded-lg shadow-lg">
<div v-if="showExportSuccess" class="fixed bottom-4 right-4 bg-blue-500 text-white px-4 py-2 shadow-lg">
Configuration exportée !
</div>
</div>
@@ -76,12 +58,12 @@
<!-- Import Modal -->
<div v-if="showImportModal" class="fixed inset-0 bg-black bg-opacity-50 flex items-center justify-center z-50 p-4">
<div class="bg-white dark:bg-gray-800 rounded-lg shadow-xl w-full max-w-md">
<div class="bg-white dark:bg-gray-800 shadow-xl w-full max-w-md">
<div class="p-6">
<h3 class="text-lg font-semibold mb-4">Importer des configurations</h3>
<textarea
v-model="importData"
class="w-full h-40 p-3 border border-gray-300 dark:border-gray-600 rounded-md dark:bg-gray-700 dark:text-white"
class="w-full h-40 p-3 border border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:text-white"
placeholder="Collez ici le JSON des configurations à importer..."></textarea>
<div class="flex justify-end space-x-3 mt-4">
@@ -93,7 +75,7 @@
<button
@click="handleImport"
:disabled="importing || !importData.trim()"
class="px-4 py-2 bg-blue-600 hover:bg-blue-700 disabled:bg-blue-400 text-white rounded-md">
class="px-4 py-2 bg-blue-600 hover:bg-blue-700 disabled:bg-blue-400 text-white">
{{ importing ? 'Import...' : 'Importer' }}
</button>
</div>
@@ -109,10 +91,11 @@ import {
ArrowPathIcon,
ArrowUpTrayIcon,
ExclamationTriangleIcon,
HeartIcon,
PlusIcon
} from '@heroicons/vue/24/outline';
import { storeToRefs } from 'pinia';
import { computed, onMounted, ref } from 'vue';
import { computed, onMounted, onUnmounted, ref } from 'vue';
import { useRouter } from 'vue-router';
import Toolbar from '../../../../shared/components/ui/Toolbar.vue';
import { useContentSourceStore } from '../../application/store/contentSourceStore';
@@ -126,9 +109,13 @@ const {
loadingSources,
sourcesError,
importing,
exporting
exporting,
checkingHealth,
} = storeToRefs(contentSourceStore);
// Mercure — écoute des mises à jour health
let mercureEventSource = null;
// Local state
const showImportModal = ref(false);
const showExportSuccess = ref(false);
@@ -138,40 +125,45 @@ const importData = ref('');
// Load sources on mount and clear current source
onMounted(async () => {
try {
contentSourceStore.clearCurrentSource(); // Clear any previously loaded source
contentSourceStore.clearErrors(); // Clear any previous errors
contentSourceStore.clearCurrentSource();
contentSourceStore.clearErrors();
await contentSourceStore.loadSources();
} catch (error) {
console.error('Erreur lors du chargement des sources:', error);
}
// Écoute Mercure pour les mises à jour de health status
const url = new URL('/.well-known/mercure', window.location.href);
sources.value.forEach(source => {
url.searchParams.append('topic', `scrapers/health/${source.id}`);
});
mercureEventSource = new EventSource(url.toString());
mercureEventSource.onmessage = (event) => {
try {
const data = JSON.parse(event.data);
contentSourceStore.updateSourceHealth(data.sourceId, data.status, data.error);
} catch (e) {
console.error('Erreur parsing Mercure event:', e);
}
};
});
onUnmounted(() => {
mercureEventSource?.close();
});
// Toolbar configuration
const toolbarConfig = computed(() => ({
leftSection: [
{
icon: ArrowPathIcon,
label: 'Actualiser',
type: 'button',
onClick: () => contentSourceStore.loadSources(),
active: loadingSources.value
}
{ type: 'label', text: 'Scrapers', class: 'text-sm font-medium' },
],
rightSection: [
{
icon: ArrowDownTrayIcon,
label: 'Exporter',
type: 'button',
onClick: handleExport,
disabled: exporting.value
},
{
icon: ArrowUpTrayIcon,
label: 'Importer',
type: 'button',
onClick: () => showImportModal.value = true
}
]
{ type: 'button', icon: ArrowPathIcon, label: 'Actualiser', onClick: () => contentSourceStore.loadSources(), disabled: loadingSources.value },
{ type: 'button', icon: HeartIcon, label: 'Tester tous', onClick: handleCheckAllHealth, disabled: checkingHealth.value },
{ type: 'button', icon: ArrowDownTrayIcon, label: 'Exporter', onClick: handleExport, disabled: exporting.value },
{ type: 'button', icon: ArrowUpTrayIcon, label: 'Importer', onClick: () => showImportModal.value = true },
],
}));
// Actions
@@ -190,6 +182,14 @@ const openSourceLink = (url) => {
window.open(url, '_blank');
};
async function handleCheckAllHealth() {
try {
await contentSourceStore.checkAllHealth();
} catch (error) {
console.error('Erreur lors du health check:', error);
}
}
async function handleExport() {
try {
const exportData = await contentSourceStore.exportSources();

View File

@@ -3,43 +3,36 @@
<Toolbar :config="toolbarConfig" />
<div class="overflow-y-auto flex-1">
<div class="container mx-auto px-4 py-6">
<!-- Back Navigation -->
<div class="mb-6">
<button
@click="goBack"
class="flex items-center space-x-2 text-gray-600 dark:text-gray-400 hover:text-gray-900 dark:hover:text-gray-200 transition-colors">
<ArrowLeftIcon class="w-5 h-5" />
<span>Retour aux configurations</span>
</button>
</div>
<!-- Loading State -->
<div v-if="loadingCurrentSource" class="flex justify-center py-12">
<div class="animate-spin rounded-full h-12 w-12 border-b-2 border-blue-500"></div>
</div>
<!-- Error State -->
<div v-else-if="currentSourceError" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 rounded-lg p-4 mb-6">
<div class="flex items-center">
<ExclamationTriangleIcon class="w-5 h-5 text-red-400 mr-2" />
<p class="text-red-800 dark:text-red-200">{{ currentSourceError }}</p>
<div class="px-6 py-8">
<section class="border-t border-gray-200 dark:border-gray-700 pt-6">
<!-- Loading State -->
<div v-if="loadingCurrentSource" class="flex justify-center py-12">
<div class="animate-spin h-12 w-12 border-b-2 border-blue-500"></div>
</div>
</div>
<!-- Form -->
<div v-else class="max-w-4xl mx-auto">
<ContentSourceForm
:source="currentSource"
:saving="saving"
:error="saveError"
@submit="handleSubmit"
@test="handleTest" />
</div>
<!-- Error State -->
<div v-else-if="currentSourceError" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-4 mb-6">
<div class="flex items-center">
<ExclamationTriangleIcon class="w-5 h-5 text-red-400 mr-2" />
<p class="text-red-800 dark:text-red-200">{{ currentSourceError }}</p>
</div>
</div>
<!-- Form -->
<div v-else>
<ContentSourceForm
ref="formRef"
:source="currentSource"
:saving="saving"
:error="saveError"
@submit="handleSubmit"
@test="handleTest" />
</div>
</section>
<!-- Test Results Modal -->
<div v-if="showTestResults" class="fixed inset-0 bg-black bg-opacity-50 flex items-center justify-center z-50 p-4">
<div class="bg-white dark:bg-gray-800 rounded-lg shadow-xl w-full max-w-4xl max-h-[80vh] overflow-hidden">
<div class="bg-white dark:bg-gray-800 shadow-xl w-full max-w-4xl max-h-[80vh] overflow-hidden">
<div class="p-6 border-b border-gray-200 dark:border-gray-600">
<div class="flex justify-between items-center">
<h3 class="text-lg font-semibold">Résultats du test</h3>
@@ -54,7 +47,7 @@
<div class="p-6 overflow-y-auto">
<!-- Loading state during test -->
<div v-if="testingConfiguration" class="flex items-center justify-center py-8">
<div class="animate-spin rounded-full h-8 w-8 border-b-2 border-blue-500 mr-3"></div>
<div class="animate-spin h-8 w-8 border-b-2 border-blue-500 mr-3"></div>
<span class="text-gray-600">Test en cours...</span>
</div>
@@ -65,7 +58,7 @@
<span class="font-medium">Test réussi !</span>
</div>
<div class="bg-green-50 dark:bg-green-900 border border-green-200 dark:border-green-700 rounded-lg p-4">
<div class="bg-green-50 dark:bg-green-900 border border-green-200 dark:border-green-700 p-4">
<div class="grid grid-cols-2 gap-4 text-sm">
<div>
<span class="font-medium text-green-800 dark:text-green-200">URL testée:</span>
@@ -92,10 +85,11 @@
<img
:src="imageUrl"
:alt="`Image ${index + 1}`"
class="w-full h-32 object-cover rounded border border-gray-200 dark:border-gray-600"
class="w-full h-32 object-cover border border-gray-200 dark:border-gray-600"
referrerpolicy="no-referrer"
@error="handleImageError"
@load="handleImageLoad" />
<div class="absolute inset-0 bg-black bg-opacity-0 group-hover:bg-opacity-30 transition-opacity rounded flex items-center justify-center">
<div class="absolute inset-0 bg-black bg-opacity-0 group-hover:bg-opacity-30 transition-opacity flex items-center justify-center">
<span class="text-white opacity-0 group-hover:opacity-100 text-sm font-medium">
Page {{ index + 1 }}
</span>
@@ -107,7 +101,7 @@
</p>
</div>
<div v-else class="bg-yellow-50 dark:bg-yellow-900 border border-yellow-200 dark:border-yellow-700 rounded-lg p-4">
<div v-else class="bg-yellow-50 dark:bg-yellow-900 border border-yellow-200 dark:border-yellow-700 p-4">
<div class="flex items-center">
<ExclamationTriangleIcon class="w-5 h-5 text-yellow-400 mr-2" />
<p class="text-yellow-800 dark:text-yellow-200">
@@ -125,7 +119,7 @@
<span class="font-medium">Test échoué</span>
</div>
<div class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 rounded-lg p-4 mb-4">
<div class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-4 mb-4">
<div class="text-sm text-red-800 dark:text-red-200">
<div><strong>URL testée:</strong> {{ testResults.testedUrl || 'N/A' }}</div>
<div><strong>Type de scraping:</strong> {{ testResults.scrapingType || 'N/A' }}</div>
@@ -138,14 +132,14 @@
<div
v-for="(error, index) in testResults.errors"
:key="index"
class="bg-red-100 dark:bg-red-800 border-l-4 border-red-400 p-4 rounded">
class="bg-red-100 dark:bg-red-800 border-l-4 border-red-400 p-4">
<div class="flex items-start">
<div class="flex-shrink-0">
<ExclamationTriangleIcon class="w-5 h-5 text-red-400" />
</div>
<div class="ml-3 flex-1">
<div class="flex items-center mb-1">
<span class="inline-flex items-center px-2 py-1 rounded-full text-xs font-medium bg-red-200 text-red-800 dark:bg-red-700 dark:text-red-200 mr-2">
<span class="inline-flex items-center px-2 py-1 text-xs font-medium bg-red-200 text-red-800 dark:bg-red-700 dark:text-red-200 mr-2">
{{ formatErrorType(error.type) }}
</span>
<span class="text-sm font-medium text-red-800 dark:text-red-200">
@@ -155,7 +149,7 @@
<p class="text-sm text-red-700 dark:text-red-300 mb-2">
{{ error.message }}
</p>
<div class="bg-red-50 dark:bg-red-900 rounded p-2">
<div class="bg-red-50 dark:bg-red-900 p-2">
<p class="text-xs text-red-600 dark:text-red-400">
<strong>Suggestion :</strong> {{ error.suggestion }}
</p>
@@ -166,7 +160,7 @@
</div>
<!-- Generic Error -->
<div v-else-if="testResults.error" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 rounded p-3">
<div v-else-if="testResults.error" class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-3">
<code class="text-sm text-red-800 dark:text-red-200">
{{ testResults.error }}
</code>
@@ -177,11 +171,20 @@
</div>
<!-- Success Message -->
<div v-if="showSuccessMessage" class="fixed bottom-4 right-4 bg-green-500 text-white px-4 py-2 rounded-lg shadow-lg">
<div v-if="showSuccessMessage" class="fixed bottom-4 right-4 bg-green-500 text-white px-4 py-2 shadow-lg">
Configuration {{ isEditing ? 'mise à jour' : 'créée' }} avec succès !
</div>
</div>
</div>
<!-- Delete Modal -->
<ContentSourceDeleteModal
:is-open="isDeleteModalOpen"
:source="currentSource"
:is-loading="isDeleting"
:error="deleteError"
@close="isDeleteModalOpen = false"
@confirm="confirmDeleteSource" />
</div>
</template>
@@ -190,6 +193,8 @@ import {
ArrowLeftIcon,
CheckCircleIcon,
ExclamationTriangleIcon,
PencilSquareIcon,
TrashIcon,
XCircleIcon,
XMarkIcon
} from '@heroicons/vue/24/outline';
@@ -199,6 +204,7 @@ import { useRoute, useRouter } from 'vue-router';
import Toolbar from '../../../../shared/components/ui/Toolbar.vue';
import { useContentSourceStore } from '../../application/store/contentSourceStore';
import { ApiContentSourceRepository } from '../../infrastructure/api/apiContentSourceRepository';
import ContentSourceDeleteModal from '../components/ContentSourceDeleteModal.vue';
import ContentSourceForm from '../components/ContentSourceForm.vue';
const route = useRoute();
@@ -214,11 +220,17 @@ const {
saveError
} = storeToRefs(contentSourceStore);
// Form ref
const formRef = ref(null);
// Local state
const showTestResults = ref(false);
const showSuccessMessage = ref(false);
const testResults = ref({});
const testingConfiguration = ref(false);
const isDeleteModalOpen = ref(false);
const isDeleting = ref(false);
const deleteError = ref(null);
const isEditing = computed(() => !!route.params.id);
@@ -233,16 +245,19 @@ onMounted(async () => {
});
// Toolbar configuration
const toolbarConfig = {
leftSection: [],
rightSection: []
};
const toolbarConfig = computed(() => ({
leftSection: [
{ type: 'button', icon: ArrowLeftIcon, label: 'Retour', onClick: () => router.push({ name: 'scrapper-configurations' }) },
{ type: 'divider' },
{ type: 'label', text: isEditing.value ? 'Modifier la configuration' : 'Nouvelle configuration', class: 'text-sm font-medium' },
],
rightSection: [
...(isEditing.value ? [{ type: 'button', icon: TrashIcon, label: 'Supprimer', onClick: () => { isDeleteModalOpen.value = true; }, class: 'text-red-600 hover:text-red-700' }, { type: 'divider' }] : []),
{ type: 'button', icon: PencilSquareIcon, label: isEditing.value ? 'Mettre à jour' : 'Créer', onClick: () => formRef.value?.submitForm(), disabled: saving.value },
],
}));
// Actions
const goBack = () => {
router.push({ name: 'scrapper-configurations' });
};
const handleSubmit = async (formData) => {
try {
if (isEditing.value) {
@@ -279,6 +294,11 @@ const handleTest = async ({ configuration, testData }) => {
testResults.value = {};
try {
// Persister testSlug + testChapterNumber avant de lancer le test
if (isEditing.value) {
await contentSourceStore.updateSource(route.params.id, configuration);
}
// Préparer les données selon le format de l'API
const testConfiguration = {
baseUrl: configuration.baseUrl,
@@ -323,6 +343,21 @@ const handleImageLoad = (event) => {
event.target.style.display = 'block';
};
const confirmDeleteSource = async () => {
isDeleting.value = true;
deleteError.value = null;
try {
await contentSourceStore.deleteSource(route.params.id);
isDeleteModalOpen.value = false;
await router.push({ name: 'scrapper-configurations' });
} catch (error) {
deleteError.value = error.message;
} finally {
isDeleting.value = false;
}
};
const formatErrorType = (type) => {
const typeMap = {
'selector_error': 'Erreur sélecteur',

View File

@@ -0,0 +1,110 @@
import { defineStore } from 'pinia';
import { ApiJobRepository } from '../../../activity/infrastructure/api/ApiJobRepository';
const jobRepository = new ApiJobRepository();
// Statuts disponibles par filtre
const STATUS_MAP = {
failed: ['failed'],
completed: ['completed'],
all: ['failed', 'completed'],
};
export const useLogsStore = defineStore('logs', {
state: () => ({
logs: [],
loading: false,
error: null,
currentPage: 1,
totalPages: 0,
total: 0,
limit: 50,
hasNextPage: false,
hasPreviousPage: false,
sortBy: 'createdAt',
sortOrder: 'DESC',
statusFilter: 'failed', // 'failed' | 'completed' | 'all'
}),
getters: {
isLoading: state => state.loading,
hasError: state => !!state.error,
},
actions: {
async loadLogs(page = null) {
this.loading = true;
this.error = null;
try {
const collection = await jobRepository.getJobs({
page: page || this.currentPage,
limit: this.limit,
sortBy: this.sortBy,
sortOrder: this.sortOrder,
status: STATUS_MAP[this.statusFilter],
type: 'scraping_job',
});
this.logs = collection.items;
this.currentPage = collection.page;
this.total = collection.total;
this.hasNextPage = collection.hasNextPage;
this.hasPreviousPage = collection.hasPreviousPage;
this.totalPages = Math.ceil(this.total / this.limit);
} catch (error) {
this.error = error.message;
} finally {
this.loading = false;
}
},
async goToPage(page) {
if (page >= 1 && page <= this.totalPages && page !== this.currentPage) {
this.currentPage = page;
await this.loadLogs(page);
}
},
async updateSort(sortBy, sortOrder) {
this.sortBy = sortBy;
this.sortOrder = sortOrder;
this.currentPage = 1;
await this.loadLogs(1);
},
async setStatusFilter(filter) {
this.statusFilter = filter;
this.currentPage = 1;
await this.loadLogs(1);
},
async deleteLog(id) {
try {
await jobRepository.deleteJob(id);
this.logs = this.logs.filter(log => log.id !== id);
this.total = Math.max(0, this.total - 1);
this.totalPages = Math.ceil(this.total / this.limit);
} catch (error) {
this.error = error.message;
}
},
async deleteAllLogs() {
this.loading = true;
this.error = null;
try {
await jobRepository.deleteJobs({
status: STATUS_MAP[this.statusFilter].join(','),
type: 'scraping_job',
});
await this.loadLogs(1);
} catch (error) {
this.error = error.message;
} finally {
this.loading = false;
}
},
},
});

View File

@@ -0,0 +1,131 @@
<template>
<div class="border-t border-gray-200 dark:border-gray-700 py-4 px-6">
<!-- Ligne 1 : Titre manga + chapitre + badge statut + date + bouton supprimer -->
<div class="flex items-start justify-between gap-4">
<div class="flex items-baseline gap-2 min-w-0">
<span class="font-semibold text-gray-900 dark:text-gray-100 truncate">
{{ log.context?.mangaTitle ?? 'Manga inconnu' }}
</span>
<span class="text-gray-400 dark:text-gray-500 text-sm shrink-0"></span>
<span class="text-sm text-gray-600 dark:text-gray-400 shrink-0">
Chapitre {{ log.context?.chapterNumber ?? '?' }}
</span>
<span
:class="[
'px-1.5 py-0.5 text-xs font-medium shrink-0',
log.status === 'completed'
? 'bg-green-100 text-green-700 dark:bg-green-900/30 dark:text-green-400'
: 'bg-red-100 text-red-700 dark:bg-red-900/30 dark:text-red-400'
]">
{{ log.status === 'completed' ? 'Terminé' : 'Échec' }}
</span>
</div>
<div class="flex items-center gap-3 shrink-0">
<span class="text-xs text-gray-400 dark:text-gray-500">{{ formattedDate }}</span>
<button
@click="$emit('delete', log.id)"
class="text-gray-400 hover:text-red-500 dark:text-gray-500 dark:hover:text-red-400 transition-colors"
title="Supprimer ce log">
<TrashIcon class="w-4 h-4" />
</button>
</div>
</div>
<!-- Ligne 2 : Source + slug + durée -->
<div class="flex items-center justify-between mt-1 gap-4">
<div class="flex items-center gap-3 min-w-0 text-sm text-gray-500 dark:text-gray-400">
<!-- Domaine de la source (lien vers la page d'édition) -->
<RouterLink
v-if="source"
:to="{ name: 'scrapper-edit', params: { id: source.id } }"
class="flex items-center gap-1 hover:text-blue-500 dark:hover:text-blue-400 transition-colors shrink-0">
<GlobeAltIcon class="w-3.5 h-3.5" />
<span class="font-mono">{{ cleanDomain }}</span>
</RouterLink>
<span v-else class="font-mono shrink-0">
ID {{ log.context?.sourceId ?? '-' }}
</span>
<!-- Badge type de scraping -->
<span
v-if="source?.scrapingType"
:class="[
'px-1.5 py-0.5 text-xs font-medium shrink-0',
source.scrapingType === 'Javascript'
? 'bg-yellow-100 text-yellow-700 dark:bg-yellow-900/30 dark:text-yellow-400'
: 'bg-blue-100 text-blue-700 dark:bg-blue-900/30 dark:text-blue-400'
]">
{{ source.scrapingType }}
</span>
<!-- Slug utilisé -->
<span v-if="log.context?.slug" class="truncate text-gray-400 dark:text-gray-500">
slug : <span class="font-mono">{{ log.context.slug }}</span>
</span>
</div>
<span v-if="duration !== null" class="text-xs text-gray-400 dark:text-gray-500 shrink-0">
{{ duration }}
</span>
</div>
<!-- Ligne 3 : Message d'erreur -->
<div v-if="log.error" class="mt-2">
<p
:class="[
'text-sm font-mono text-red-600 dark:text-red-400',
!expanded && isLong ? 'line-clamp-1' : ''
]">
{{ log.error }}
</p>
<button
v-if="isLong"
@click="expanded = !expanded"
class="mt-1 text-xs text-gray-400 hover:text-gray-600 dark:hover:text-gray-300 transition-colors">
{{ expanded ? 'voir moins' : 'voir plus' }}
</button>
</div>
</div>
</template>
<script setup>
import { GlobeAltIcon, TrashIcon } from '@heroicons/vue/24/outline';
import { computed, ref } from 'vue';
import { RouterLink } from 'vue-router';
const props = defineProps({
log: {
type: Object,
required: true,
},
source: {
type: Object,
default: null,
},
});
defineEmits(['delete']);
const expanded = ref(false);
const isLong = computed(() => props.log.error && props.log.error.length > 120);
const cleanDomain = computed(() => {
if (!props.source?.baseUrl) return null;
return props.source.baseUrl.replace(/^(https?:\/\/)?(www\.)?/, '').replace(/\/+$/, '');
});
const formattedDate = computed(() => {
if (!props.log.createdAt) return '';
const d = new Date(props.log.createdAt);
const pad = n => String(n).padStart(2, '0');
return `${pad(d.getDate())}/${pad(d.getMonth() + 1)}/${d.getFullYear()} ${pad(d.getHours())}:${pad(d.getMinutes())}`;
});
const duration = computed(() => {
if (!props.log.startedAt || !props.log.completedAt) return null;
const ms = new Date(props.log.completedAt) - new Date(props.log.startedAt);
if (ms < 0) return null;
return `${(ms / 1000).toLocaleString('fr-FR', { maximumFractionDigits: 1 })}s`;
});
</script>

View File

@@ -0,0 +1,165 @@
<template>
<div class="flex flex-col h-full">
<Toolbar :config="toolbarConfig" />
<div class="overflow-y-auto flex-1">
<section class="border-t border-gray-200 dark:border-gray-700">
<!-- Loading -->
<div v-if="isLoading" class="flex justify-center py-12">
<div class="animate-spin h-10 w-10 border-b-2 border-blue-500 rounded-full"></div>
</div>
<!-- Error -->
<div v-else-if="hasError" class="px-6 py-8">
<div class="bg-red-50 dark:bg-red-900 border border-red-200 dark:border-red-700 p-4">
<div class="flex items-center">
<ExclamationCircleIcon class="w-5 h-5 text-red-400 mr-2 shrink-0" />
<p class="text-red-800 dark:text-red-200">{{ error }}</p>
</div>
<button
@click="logsStore.loadLogs()"
class="mt-3 px-4 py-2 bg-red-600 text-white hover:bg-red-700">
Réessayer
</button>
</div>
</div>
<!-- Empty -->
<div v-else-if="!isLoading && logs.length === 0" class="flex flex-col items-center justify-center py-20 text-gray-400 dark:text-gray-500">
<ExclamationCircleIcon class="w-12 h-12 mb-3" />
<p class="text-base">Aucune erreur de scraping</p>
</div>
<!-- List -->
<template v-else>
<LogItem
v-for="log in logs"
:key="log.id"
:log="log"
:source="getSource(log)"
@delete="handleDelete" />
</template>
</section>
<!-- Pagination -->
<Pagination
v-if="totalPages > 1"
:current-page="currentPage"
:total-pages="totalPages"
:total="total"
:limit="limit"
:has-next-page="hasNextPage"
:has-previous-page="hasPreviousPage"
@page-change="logsStore.goToPage" />
</div>
</div>
</template>
<script setup>
import { ArrowPathIcon, ExclamationCircleIcon, TrashIcon } from '@heroicons/vue/24/outline';
import { BarsArrowDownIcon } from '@heroicons/vue/24/outline';
import { storeToRefs } from 'pinia';
import { computed, onMounted } from 'vue';
import Toolbar from '../../../../shared/components/ui/Toolbar.vue';
import Pagination from '../../../../shared/components/ui/Pagination.vue';
import { useContentSourceStore } from '../../../setting/application/store/contentSourceStore';
import { useLogsStore } from '../../application/store/logsStore';
import LogItem from '../components/LogItem.vue';
const logsStore = useLogsStore();
const contentSourceStore = useContentSourceStore();
const { sources } = storeToRefs(contentSourceStore);
const {
logs,
loading: isLoading,
error,
currentPage,
totalPages,
total,
limit,
hasNextPage,
hasPreviousPage,
sortBy,
sortOrder,
statusFilter,
} = storeToRefs(logsStore);
const hasError = computed(() => !!error.value);
onMounted(() => {
logsStore.loadLogs();
contentSourceStore.loadSources();
});
function getSource(log) {
const sourceId = log.context?.sourceId;
if (!sourceId) return null;
// eslint-disable-next-line eqeqeq
return sources.value.find(s => s.id == sourceId) ?? null;
}
const isSortSelected = (by, order) => sortBy.value === by && sortOrder.value === order;
const STATUS_FILTERS = [
{ key: 'failed', label: 'Échecs' },
{ key: 'completed', label: 'Terminés' },
{ key: 'all', label: 'Tous' },
];
const toolbarConfig = computed(() => ({
leftSection: [
{ type: 'label', text: 'Logs', class: 'text-sm font-medium' },
{ type: 'label', text: `(${total.value})`, class: 'text-sm text-gray-400' },
],
rightSection: [
...STATUS_FILTERS.map(f => ({
type: 'button',
label: f.label,
active: statusFilter.value === f.key,
onClick: () => logsStore.setStatusFilter(f.key),
})),
{ type: 'divider' },
{
type: 'dropdown',
icon: BarsArrowDownIcon,
label: 'Trier',
items: [
{
label: 'Plus récent',
isSelected: isSortSelected('createdAt', 'DESC'),
onClick: () => logsStore.updateSort('createdAt', 'DESC'),
},
{
label: 'Plus ancien',
isSelected: isSortSelected('createdAt', 'ASC'),
onClick: () => logsStore.updateSort('createdAt', 'ASC'),
},
],
},
{
type: 'button',
icon: ArrowPathIcon,
label: 'Rafraîchir',
disabled: isLoading.value,
onClick: () => logsStore.loadLogs(),
},
{
type: 'button',
icon: TrashIcon,
label: 'Tout supprimer',
disabled: isLoading.value || total.value === 0,
onClick: handleDeleteAll,
},
],
}));
async function handleDelete(id) {
await logsStore.deleteLog(id);
}
async function handleDeleteAll() {
if (!confirm('Supprimer tous les logs d\'erreur ? Cette action est irréversible.')) return;
await logsStore.deleteAllLogs();
}
</script>

View File

@@ -10,6 +10,7 @@ import ChapterPage from '../domain/reader/presentation/pages/ChapterPage.vue';
import ScrapperConfigurations from '../domain/setting/presentation/pages/ScrapperConfigurations.vue';
import ScrapperEdit from '../domain/setting/presentation/pages/ScrapperEdit.vue';
import UserPreferencesPage from '../domain/setting/presentation/pages/UserPreferencesPage.vue';
import LogsPage from '../domain/system/presentation/pages/LogsPage.vue';
import Layout from '../shared/components/layout/Layout.vue';
// Placeholder component for new routes
@@ -148,8 +149,7 @@ const routes = [
{
path: '/system/logs',
name: 'system-logs',
component: PlaceholderComponent,
props: { title: 'Journaux système' }
component: LogsPage,
},
{
path: '/system/updates',

View File

@@ -17,7 +17,6 @@ framework:
command.bus:
middleware:
- validation
- doctrine_transaction
event.bus:
default_middleware: allow_no_handlers

View File

@@ -180,6 +180,13 @@ services:
tags:
- { name: messenger.message_handler, bus: command.bus }
# Scraper Health Check
App\Domain\Scraping\Domain\Contract\Repository\ContentSourceForHealthCheckInterface:
alias: App\Domain\Setting\Infrastructure\Persistence\Repository\DoctrineContentSourceForHealthCheckRepository
App\Domain\Scraping\Domain\Contract\Repository\ContentSourceHealthRepositoryInterface:
alias: App\Domain\Setting\Infrastructure\Persistence\Repository\DoctrineContentSourceForHealthCheckRepository
# Import Domain Services
App\Domain\Import\Infrastructure\Service\FilenameAnalyzer: ~

View File

@@ -0,0 +1,41 @@
<?php
declare(strict_types=1);
namespace DoctrineMigrations;
use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;
/**
* Auto-generated Migration: Please modify to your needs!
*/
final class Version20260315221706 extends AbstractMigration
{
public function getDescription(): string
{
return '';
}
public function up(Schema $schema): void
{
// this up() migration is auto-generated, please modify it to your needs
$this->addSql('ALTER TABLE content_source ADD test_slug VARCHAR(255) DEFAULT NULL');
$this->addSql('ALTER TABLE content_source ADD test_chapter_number DOUBLE PRECISION DEFAULT NULL');
$this->addSql('ALTER TABLE content_source ADD health_status VARCHAR(20) DEFAULT \'unknown\' NOT NULL');
$this->addSql('ALTER TABLE content_source ADD health_last_tested_at TIMESTAMP(0) WITHOUT TIME ZONE DEFAULT NULL');
$this->addSql('ALTER TABLE content_source ADD health_last_error TEXT DEFAULT NULL');
$this->addSql('COMMENT ON COLUMN content_source.health_last_tested_at IS \'(DC2Type:datetime_immutable)\'');
}
public function down(Schema $schema): void
{
// this down() migration is auto-generated, please modify it to your needs
$this->addSql('CREATE SCHEMA public');
$this->addSql('ALTER TABLE content_source DROP test_slug');
$this->addSql('ALTER TABLE content_source DROP test_chapter_number');
$this->addSql('ALTER TABLE content_source DROP health_status');
$this->addSql('ALTER TABLE content_source DROP health_last_tested_at');
$this->addSql('ALTER TABLE content_source DROP health_last_error');
}
}

View File

@@ -0,0 +1,7 @@
<?php
namespace App\Domain\Scraping\Application\Command;
readonly class CheckAllScrapersHealth
{
}

View File

@@ -5,7 +5,8 @@ namespace App\Domain\Scraping\Application\Command;
readonly class ScrapeChapter
{
public function __construct(
public string $chapterId
public string $chapterId,
public string $jobId
) {
}
}

View File

@@ -0,0 +1,64 @@
<?php
namespace App\Domain\Scraping\Application\CommandHandler;
use App\Domain\Scraping\Application\Command\CheckAllScrapersHealth;
use App\Domain\Scraping\Application\Command\TestScraperConfiguration;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceForHealthCheckInterface;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceHealthRepositoryInterface;
use Psr\Log\LoggerInterface;
readonly class CheckAllScrapersHealthHandler
{
public function __construct(
private ContentSourceForHealthCheckInterface $contentSourceForHealthCheckRepo,
private ContentSourceHealthRepositoryInterface $contentSourceHealthRepo,
private TestScraperConfigurationHandler $testScraperConfigurationHandler,
private LoggerInterface $logger,
) {
}
public function handle(CheckAllScrapersHealth $command): void
{
$sources = $this->contentSourceForHealthCheckRepo->getAll();
foreach ($sources as $source) {
if ($source->testSlug === null || $source->testChapterNumber === null) {
$this->logger->warning('ContentSource {id} has no test config, skipping health check.', ['id' => $source->id]);
continue;
}
try {
$this->contentSourceHealthRepo->markAsTesting($source->id);
$testUrl = str_replace(
['{slug}', '{chapterNumber}'],
[$source->testSlug, $source->testChapterNumber],
$source->chapterUrlFormat
);
$testCommand = new TestScraperConfiguration(
baseUrl: $source->baseUrl,
chapterUrlFormat: $source->chapterUrlFormat,
scrapingType: $source->scrapingType,
testUrl: $testUrl,
mangaSlug: $source->testSlug,
chapterNumber: $source->testChapterNumber,
imageSelector: $source->imageSelector,
nextPageSelector: $source->nextPageSelector,
chapterSelector: $source->chapterSelector,
);
$response = $this->testScraperConfigurationHandler->handle($testCommand);
if ($response->success) {
$this->contentSourceHealthRepo->markAsHealthy($source->id, new \DateTimeImmutable());
} else {
$firstError = $response->errors[0]['message'] ?? 'Erreur inconnue';
$this->contentSourceHealthRepo->markAsUnhealthy($source->id, new \DateTimeImmutable(), $firstError);
}
} catch (\Exception $e) {
$this->contentSourceHealthRepo->markAsUnhealthy($source->id, new \DateTimeImmutable(), $e->getMessage());
}
}
}
}

View File

@@ -13,14 +13,11 @@ use App\Domain\Shared\Domain\Event\ChapterScraped;
use App\Domain\Scraping\Domain\Event\ChapterScrapingFailed;
use App\Domain\Scraping\Domain\Event\ChapterScrapingStarted;
use App\Domain\Scraping\Domain\Model\Chapter;
use App\Domain\Scraping\Domain\Model\ScrapingJob;
use App\Domain\Scraping\Domain\Model\Source;
use App\Domain\Scraping\Domain\Model\ValueObject\ScrapingRequest;
use App\Domain\Scraping\Domain\Model\ValueObject\TempDirectory;
use App\Domain\Shared\Domain\Contract\JobRepositoryInterface;
use Ramsey\Uuid\Uuid;
use Symfony\Component\Messenger\MessageBusInterface;
use Doctrine\ORM\EntityManagerInterface;
readonly class ScrapeChapterHandler
{
@@ -33,151 +30,92 @@ readonly class ScrapeChapterHandler
private MangaRepositoryInterface $mangaRepository,
private SourceRepositoryInterface $sourceRepository,
private MessageBusInterface $eventBus,
private EntityManagerInterface $entityManager
) {
}
public function handle(ScrapeChapter $command): void
{
$job = null;
try {
// 1. Récupération du chapitre
/**@var Chapter $chapter */
$chapter = $this->chapterRepository->getById($command->chapterId);
if (!$chapter) {
throw new \InvalidArgumentException("Chapter not found with ID: {$command->chapterId}");
}
/** @var Chapter $chapter */
$chapter = $this->chapterRepository->getById($command->chapterId);
$manga = $this->mangaRepository->getById($chapter->mangaId);
// 2. Récupération du manga
$manga = $this->mangaRepository->getById($chapter->mangaId);
if (!$manga) {
throw new \InvalidArgumentException("Manga not found with ID: {$chapter->mangaId}");
}
$job = $this->jobRepository->get($command->jobId);
$job->context['chapterId'] = $command->chapterId;
$job->context['mangaTitle'] = $manga->getTitle();
$job->start();
$this->jobRepository->save($job);
// 3. Dispatch de l'événement de démarrage
$this->eventBus->dispatch(new ChapterScrapingStarted($manga->getTitle(), $chapter->chapterNumber));
$this->eventBus->dispatch(new ChapterScrapingStarted($job->id, $manga->getTitle(), $chapter->chapterNumber));
// 4. Détermination des sources à utiliser
$sources = $this->getSourcesToTry($manga);
if (empty($sources)) {
throw new \InvalidArgumentException("No sources available for scraping");
}
$sources = $this->getSourcesToTry($manga);
$slugsToTry = array_merge([$manga->getSlug()], $manga->getAlternativeSlugs());
$success = false;
$lastException = null;
// 5. Essai de scraping sur chaque source jusqu'à succès
$success = false;
$lastException = null;
foreach ($sources as $source) {
// Préparer la liste des slugs à essayer : slug principal + slugs alternatifs
$slugsToTry = array_merge([$manga->getSlug()], $manga->getAlternativeSlugs());
foreach ($slugsToTry as $slug) {
$job = new ScrapingJob(
Uuid::uuid4()->toString(),
$chapter->mangaId,
$chapter->chapterNumber,
$source->getId()->getValue()
);
// Ajout de l'ID du chapitre et du slug dans le contexte du job
$job->context['chapterId'] = $command->chapterId;
foreach ($sources as $source) {
foreach ($slugsToTry as $slug) {
try {
$job->context['sourceId'] = $source->getId()->getValue();
$job->context['slug'] = $slug;
$job->context['mangaTitle'] = $manga->getTitle();
$job->start();
$this->jobRepository->save($job);
try {
$this->entityManager->beginTransaction();
$scrapingParameters = $source->getScrappingParameters();
$scrapingParameters['chapterNumber'] = $chapter->chapterNumber;
$scrapingType = $scrapingParameters['scrapingType'] ?? 'html';
// 5. Scraping des URLs avec le slug courant
$scrapingParameters = $source->getScrappingParameters();
$scrapingParameters['chapterNumber'] = $chapter->chapterNumber;
$scrapingType = $scrapingParameters['scrapingType'] ?? 'html';
$scrapingRequest = new ScrapingRequest(
$scrapingType,
$source->buildChapterUrl($slug, $chapter->chapterNumber),
$scrapingParameters
);
$scrapingRequest = new ScrapingRequest(
$scrapingType,
$source->buildChapterUrl($slug, $chapter->chapterNumber),
$scrapingParameters
);
$scraper = $this->scraperFactory->getScraperWithFallback($scrapingType);
$scrapingResult = $scraper->scrape($scrapingRequest);
// Sélection du scraper approprié selon le type
$scraper = $this->scraperFactory->getScraperWithFallback($scrapingType);
$scrapingResult = $scraper->scrape($scrapingRequest);
$tempDir = new TempDirectory();
$downloadResults = $this->imageDownloader->downloadBatch(
$scrapingResult->getImageUrls(),
$tempDir,
$job->id
);
// 6. Téléchargement des images
$tempDir = new TempDirectory();
$downloadResults = $this->imageDownloader->downloadBatch(
$scrapingResult->getImageUrls(),
$tempDir,
$job->id
);
$localPaths = array_map(fn ($r) => $r->getLocalPath(), $downloadResults);
$pagesDirectory = $this->imageStorage->storeChapterImages($command->chapterId, $localPaths);
$pageCount = count($downloadResults);
// 7. Stockage des images individuelles
$localPaths = array_map(fn ($r) => $r->getLocalPath(), $downloadResults);
$pagesDirectory = $this->imageStorage->storeChapterImages($command->chapterId, $localPaths);
$pageCount = count($downloadResults);
$job->complete();
$this->jobRepository->save($job);
$job->complete();
$this->jobRepository->save($job);
$this->eventBus->dispatch(new ChapterScraped($job->id, $command->chapterId, $pagesDirectory, $pageCount));
$tempDir->cleanup();
$this->entityManager->commit();
$this->eventBus->dispatch(new ChapterScraped($job->id, $command->chapterId, $pagesDirectory, $pageCount));
// 8. Nettoyage
$tempDir->cleanup();
// Scraping réussi, pas besoin d'essayer d'autres slugs ni d'autres sources
$success = true;
break;
} catch (\Exception $e) {
dump('EXCEPTION for source ' . $source->getName() . ' with slug ' . $slug . ': ' . $e->getMessage());
$this->entityManager->rollback();
if (isset($job)) {
$job->fail($e->getMessage());
$this->jobRepository->save($job);
}
$lastException = $e;
// Continuer avec le slug suivant pour cette source
}
}
// Si le scraping a réussi avec un des slugs, sortir de la boucle des sources
if ($success) {
$success = true;
break;
} catch (\Exception $e) {
$lastException = $e;
}
}
// Si toutes les sources ont échoué
if (!$success) {
$errorMessage = $lastException ? $lastException->getMessage() : "Failed to scrape chapter from all available sources";
$this->eventBus->dispatch(new ChapterScrapingFailed($chapter->mangaId, $chapter->chapterNumber, $errorMessage));
if ($success) {
break;
}
}
} catch (\Exception $e) {
if (isset($job)) {
$job->fail($e->getMessage());
$this->jobRepository->save($job);
}
$this->eventBus->dispatch(new ChapterScrapingFailed($chapter->mangaId ?? 'unknown', $chapter->chapterNumber ?? 'unknown', $e->getMessage()));
if (!$success) {
$errorMessage = $lastException?->getMessage() ?? 'Failed to scrape chapter from all available sources';
$job->fail($errorMessage);
$this->jobRepository->save($job);
$this->eventBus->dispatch(new ChapterScrapingFailed($job->id, $chapter->mangaId, $chapter->chapterNumber, $errorMessage));
}
}
/**
* Détermine les sources à utiliser pour le scraping en fonction des préférences du manga
*
* @param \App\Domain\Scraping\Domain\Model\Manga $manga
* @return Source[]
*/
private function getSourcesToTry(\App\Domain\Scraping\Domain\Model\Manga $manga): array
{
// Si le manga a des sources préférées, les utiliser
if ($manga->hasPreferredSources()) {
$preferredSources = [];
foreach ($manga->getPreferredSources() as $sourceId) {
@@ -186,7 +124,6 @@ readonly class ScrapeChapterHandler
$preferredSources[] = $source;
}
// Limiter à 3 sources préférées maximum
if (count($preferredSources) >= 3) {
break;
}
@@ -197,7 +134,6 @@ readonly class ScrapeChapterHandler
}
}
// Sinon, utiliser toutes les sources disponibles
return $this->sourceRepository->getAll();
}
}

View File

@@ -0,0 +1,11 @@
<?php
namespace App\Domain\Scraping\Domain\Contract\Repository;
use App\Domain\Scraping\Domain\Model\ValueObject\ContentSourceHealthCheckData;
interface ContentSourceForHealthCheckInterface
{
/** @return ContentSourceHealthCheckData[] */
public function getAll(): array;
}

View File

@@ -0,0 +1,12 @@
<?php
namespace App\Domain\Scraping\Domain\Contract\Repository;
interface ContentSourceHealthRepositoryInterface
{
public function markAsTesting(int $sourceId): void;
public function markAsHealthy(int $sourceId, \DateTimeImmutable $testedAt): void;
public function markAsUnhealthy(int $sourceId, \DateTimeImmutable $testedAt, string $error): void;
}

View File

@@ -5,12 +5,18 @@ namespace App\Domain\Scraping\Domain\Event;
readonly class ChapterScrapingFailed
{
public function __construct(
private string $jobId,
private string $mangaId,
private string $chapterNumber,
private string $reason
) {
}
public function getJobId(): string
{
return $this->jobId;
}
public function getMangaId(): string
{
return $this->mangaId;

View File

@@ -5,11 +5,17 @@ namespace App\Domain\Scraping\Domain\Event;
class ChapterScrapingStarted
{
public function __construct(
private readonly string $jobId,
private readonly string $mangaTitle,
private readonly float $chapterNumber,
) {
}
public function getJobId(): string
{
return $this->jobId;
}
public function getMangaTitle(): string
{
return $this->mangaTitle;

View File

@@ -8,9 +8,9 @@ class ScrapingJob extends Job
{
public function __construct(
string $id,
string $mangaId,
float $chapterNumber,
string $sourceId
?string $mangaId = null,
?float $chapterNumber = null,
?string $sourceId = null
) {
parent::__construct($id, 'scraping_job');
$this->maxAttempts = 1;

View File

@@ -0,0 +1,19 @@
<?php
namespace App\Domain\Scraping\Domain\Model\ValueObject;
readonly class ContentSourceHealthCheckData
{
public function __construct(
public int $id,
public string $baseUrl,
public string $chapterUrlFormat,
public string $scrapingType,
public ?string $imageSelector,
public ?string $nextPageSelector,
public ?string $chapterSelector,
public ?string $testSlug,
public ?float $testChapterNumber,
) {
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace App\Domain\Scraping\Infrastructure\ApiPlatform\Resource;
use ApiPlatform\Metadata\ApiResource;
use ApiPlatform\Metadata\Post;
use App\Domain\Scraping\Infrastructure\ApiPlatform\State\Processor\CheckAllScrapersHealthStateProcessor;
#[ApiResource(
shortName: 'Scraping',
operations: [
new Post(
uriTemplate: '/scraping/check-all-health',
processor: CheckAllScrapersHealthStateProcessor::class,
output: false,
status: 202,
description: 'Déclenche le test de santé de tous les scrapers configurés avec testSlug',
),
]
)]
class CheckAllScrapersHealthResource
{
}

View File

@@ -0,0 +1,23 @@
<?php
namespace App\Domain\Scraping\Infrastructure\ApiPlatform\State\Processor;
use ApiPlatform\Metadata\Operation;
use ApiPlatform\State\ProcessorInterface;
use App\Domain\Scraping\Application\Command\CheckAllScrapersHealth;
use App\Domain\Scraping\Application\CommandHandler\CheckAllScrapersHealthHandler;
readonly class CheckAllScrapersHealthStateProcessor implements ProcessorInterface
{
public function __construct(
private CheckAllScrapersHealthHandler $handler,
) {
}
public function process(mixed $data, Operation $operation, array $uriVariables = [], array $context = []): null
{
$this->handler->handle(new CheckAllScrapersHealth());
return null;
}
}

View File

@@ -5,13 +5,20 @@ namespace App\Domain\Scraping\Infrastructure\ApiPlatform\State\Processor;
use ApiPlatform\Metadata\Operation;
use ApiPlatform\State\ProcessorInterface;
use App\Domain\Scraping\Application\Command\ScrapeChapter;
use App\Domain\Scraping\Domain\Model\ScrapingJob;
use App\Domain\Scraping\Infrastructure\ApiPlatform\Dto\ScrapeChapterRequest;
use App\Domain\Shared\Domain\Contract\JobRepositoryInterface;
use Ramsey\Uuid\Uuid;
use Symfony\Component\Mercure\HubInterface;
use Symfony\Component\Mercure\Update;
use Symfony\Component\Messenger\MessageBusInterface;
final class ScrapeChapterStateProcessor implements ProcessorInterface
{
public function __construct(
private readonly MessageBusInterface $commandBus
private readonly MessageBusInterface $commandBus,
private readonly JobRepositoryInterface $jobRepository,
private readonly HubInterface $hub,
) {
}
@@ -20,10 +27,25 @@ final class ScrapeChapterStateProcessor implements ProcessorInterface
*/
public function process(mixed $data, Operation $operation, array $uriVariables = [], array $context = []): void
{
$this->commandBus->dispatch(
new ScrapeChapter(
$data->chapterId
)
);
$jobId = Uuid::uuid4()->toString();
$job = new ScrapingJob($jobId);
$job->context['chapterId'] = $data->chapterId;
$this->jobRepository->save($job);
$this->hub->publish(new Update(
'jobs/activity',
json_encode([
'type' => 'job.created',
'id' => $job->id,
'type_job' => $job->type,
'status' => $job->status->value,
'createdAt' => $job->createdAt->format('c'),
'context' => $job->context,
'attempts' => $job->attempts,
'maxAttempts' => $job->maxAttempts,
])
));
$this->commandBus->dispatch(new ScrapeChapter($data->chapterId, $jobId));
}
}

View File

@@ -5,6 +5,7 @@ namespace App\Domain\Scraping\Infrastructure\EventSubscriber;
use App\Domain\Shared\Domain\Event\ChapterScraped;
use App\Domain\Scraping\Domain\Event\ChapterScrapingFailed;
use App\Domain\Scraping\Domain\Event\ChapterScrapingStarted;
use App\Domain\Scraping\Domain\Event\PageScrapingProgressed;
use App\Domain\Scraping\Domain\Contract\Repository\ChapterRepositoryInterface;
use App\Domain\Shared\Domain\Contract\JobRepositoryInterface;
use App\Domain\Shared\Domain\Contract\NotificationInterface;
@@ -30,14 +31,32 @@ class ScrapingEventSubscriber implements EventSubscriberInterface
return [];
}
#[AsMessageHandler]
public function onPageScrapingProgressed(PageScrapingProgressed $event): void
{
$progress = (int) round($event->getProgress()->getPercentage());
$update = new Update(
'jobs/activity',
json_encode([
'type' => 'job.progress_updated',
'jobId' => $event->getJobId(),
'progress' => $progress,
])
);
$this->hub->publish($update);
}
#[AsMessageHandler]
public function onChapterScrapingStarted(ChapterScrapingStarted $event): void
{
$chapterNumber = $event->getChapterNumber();
$mangaTitle = $event->getMangaTitle();
$this->hub->publish(new Update(
'jobs/activity',
json_encode(['type' => 'job.status_changed', 'jobId' => $event->getJobId(), 'status' => 'in_progress'])
));
$this->notification->sendInfo(
sprintf('Scraping du chapitre %s de "%s" démarré', $chapterNumber, $mangaTitle)
sprintf('Scraping du chapitre %s de "%s" démarré', $event->getChapterNumber(), $event->getMangaTitle())
);
}
@@ -84,6 +103,11 @@ class ScrapingEventSubscriber implements EventSubscriberInterface
$update = new Update($topics, json_encode($data));
$this->hub->publish($update);
$this->hub->publish(new Update(
'jobs/activity',
json_encode(['type' => 'job.status_changed', 'jobId' => $jobId, 'status' => 'completed'])
));
$mangaTitle = $job->context['mangaTitle'] ?? 'manga inconnu';
$this->notification->sendSuccess(
sprintf('Chapitre %s de "%s" scrappé avec succès', $chapter->chapterNumber, $mangaTitle)
@@ -93,6 +117,11 @@ class ScrapingEventSubscriber implements EventSubscriberInterface
#[AsMessageHandler]
public function onChapterScrapingFailed(ChapterScrapingFailed $event): void
{
$this->hub->publish(new Update(
'jobs/activity',
json_encode(['type' => 'job.status_changed', 'jobId' => $event->getJobId(), 'status' => 'failed'])
));
$this->logger->info('ChapterScrapingFailed reçu pour mangaId: ' . $event->getMangaId() . ', chapter: ' . $event->getChapterNumber());
$data = [

View File

@@ -20,7 +20,14 @@ readonly class ImageDownloader implements ImageDownloaderInterface
public function download(string $url, string $destination): void
{
$response = $this->httpClient->request('GET', $url);
$urlParts = parse_url($url);
$referer = ($urlParts['scheme'] ?? 'https') . '://' . ($urlParts['host'] ?? '');
$response = $this->httpClient->request('GET', $url, [
'headers' => [
'Referer' => $referer,
],
]);
$contentType = $response->getHeaders()['content-type'][0] ?? '';
if (!str_starts_with($contentType, 'image/')) {

View File

@@ -0,0 +1,11 @@
<?php
namespace App\Domain\Setting\Application\Command;
readonly class DeleteContentSourceCommand
{
public function __construct(
public int $id
) {
}
}

View File

@@ -12,6 +12,8 @@ readonly class UpsertContentSourceCommand
public ?string $imageSelector = null,
public ?string $nextPageSelector = null,
public ?string $chapterSelector = null,
public ?string $testSlug = null,
public ?float $testChapterNumber = null,
) {
}
}

View File

@@ -0,0 +1,26 @@
<?php
namespace App\Domain\Setting\Application\CommandHandler;
use App\Domain\Setting\Application\Command\DeleteContentSourceCommand;
use App\Domain\Setting\Domain\Contract\Repository\ContentSourceRepositoryInterface;
use App\Domain\Setting\Domain\Exception\ContentSourceNotFoundException;
readonly class DeleteContentSourceCommandHandler
{
public function __construct(
private ContentSourceRepositoryInterface $contentSourceRepository
) {
}
public function handle(DeleteContentSourceCommand $command): void
{
$contentSource = $this->contentSourceRepository->findById($command->id);
if (!$contentSource) {
throw new ContentSourceNotFoundException($command->id);
}
$this->contentSourceRepository->delete($contentSource);
}
}

View File

@@ -26,6 +26,8 @@ readonly class UpsertContentSourceCommandHandler
imageSelector: $command->imageSelector,
nextPageSelector: $command->nextPageSelector,
chapterSelector: $command->chapterSelector,
testSlug: $command->testSlug,
testChapterNumber: $command->testChapterNumber,
);
$this->contentSourceRepository->save($contentSource);
}
@@ -38,6 +40,8 @@ readonly class UpsertContentSourceCommandHandler
imageSelector: $command->imageSelector,
nextPageSelector: $command->nextPageSelector,
chapterSelector: $command->chapterSelector,
testSlug: $command->testSlug,
testChapterNumber: $command->testChapterNumber,
);
$this->contentSourceRepository->save($contentSource);
}

View File

@@ -15,6 +15,11 @@ readonly class ContentSourceResponse
public ?string $nextPageSelector,
public ?string $chapterSelector,
public string $cleanBaseUrl,
public ?string $testSlug = null,
public ?float $testChapterNumber = null,
public string $healthStatus = 'unknown',
public ?\DateTimeImmutable $healthLastTestedAt = null,
public ?string $healthLastError = null,
) {
}
@@ -29,6 +34,11 @@ readonly class ContentSourceResponse
nextPageSelector: $contentSource->getNextPageSelector(),
chapterSelector: $contentSource->getChapterSelector(),
cleanBaseUrl: $contentSource->getCleanBaseUrl(),
testSlug: $contentSource->getTestSlug(),
testChapterNumber: $contentSource->getTestChapterNumber(),
healthStatus: $contentSource->getHealthStatus(),
healthLastTestedAt: $contentSource->getHealthLastTestedAt(),
healthLastError: $contentSource->getHealthLastError(),
);
}
}

View File

@@ -12,6 +12,11 @@ final class ContentSource
private ?string $imageSelector = null,
private ?string $nextPageSelector = null,
private ?string $chapterSelector = null,
private ?string $testSlug = null,
private ?float $testChapterNumber = null,
private string $healthStatus = 'unknown',
private ?\DateTimeImmutable $healthLastTestedAt = null,
private ?string $healthLastError = null,
) {
}
@@ -50,6 +55,44 @@ final class ContentSource
return $this->chapterSelector;
}
public function getTestSlug(): ?string
{
return $this->testSlug;
}
public function getTestChapterNumber(): ?float
{
return $this->testChapterNumber;
}
public function getHealthStatus(): string
{
return $this->healthStatus;
}
public function getHealthLastTestedAt(): ?\DateTimeImmutable
{
return $this->healthLastTestedAt;
}
public function getHealthLastError(): ?string
{
return $this->healthLastError;
}
public function updateTestConfig(?string $testSlug, ?float $testChapterNumber): void
{
$this->testSlug = $testSlug;
$this->testChapterNumber = $testChapterNumber;
}
public function updateHealthStatus(string $status, ?\DateTimeImmutable $testedAt = null, ?string $error = null): void
{
$this->healthStatus = $status;
$this->healthLastTestedAt = $testedAt;
$this->healthLastError = $error;
}
public function updateId(int $id): void
{
$this->id = $id;
@@ -71,6 +114,8 @@ final class ContentSource
?string $imageSelector = null,
?string $nextPageSelector = null,
?string $chapterSelector = null,
?string $testSlug = null,
?float $testChapterNumber = null,
): self {
return new self(
id: null,
@@ -80,6 +125,8 @@ final class ContentSource
imageSelector: $imageSelector,
nextPageSelector: $nextPageSelector,
chapterSelector: $chapterSelector,
testSlug: $testSlug,
testChapterNumber: $testChapterNumber,
);
}
@@ -90,6 +137,8 @@ final class ContentSource
?string $imageSelector = null,
?string $nextPageSelector = null,
?string $chapterSelector = null,
?string $testSlug = null,
?float $testChapterNumber = null,
): void {
$this->baseUrl = $baseUrl;
$this->chapterUrlFormat = $chapterUrlFormat;
@@ -97,5 +146,7 @@ final class ContentSource
$this->imageSelector = $imageSelector;
$this->nextPageSelector = $nextPageSelector;
$this->chapterSelector = $chapterSelector;
$this->testSlug = $testSlug;
$this->testChapterNumber = $testChapterNumber;
}
}

View File

@@ -0,0 +1,50 @@
<?php
namespace App\Domain\Setting\Infrastructure\ApiPlatform\Resource;
use ApiPlatform\Metadata\ApiResource;
use ApiPlatform\Metadata\Delete;
use App\Domain\Setting\Infrastructure\ApiPlatform\State\Processor\DeleteContentSourceStateProcessor;
use App\Domain\Setting\Infrastructure\ApiPlatform\State\Provider\DeleteContentSourceStateProvider;
#[ApiResource(
shortName: 'ContentSource',
operations: [
new Delete(
uriTemplate: '/content-sources/{id}',
provider: DeleteContentSourceStateProvider::class,
processor: DeleteContentSourceStateProcessor::class,
name: 'delete_content_source',
openapiContext: [
'summary' => 'Delete a content source',
'description' => 'Permanently deletes a content source',
'parameters' => [
[
'name' => 'id',
'in' => 'path',
'required' => true,
'schema' => [
'type' => 'integer'
],
'description' => 'The content source ID'
]
],
'responses' => [
'204' => [
'description' => 'Content source successfully deleted'
],
'404' => [
'description' => 'Content source not found'
]
]
]
)
]
)]
class DeleteContentSourceResource
{
public function __construct(
public int $id
) {
}
}

View File

@@ -30,6 +30,11 @@ class GetContentSourceResource
public readonly ?string $nextPageSelector,
public readonly ?string $chapterSelector,
public readonly string $cleanBaseUrl,
public readonly ?string $testSlug = null,
public readonly ?float $testChapterNumber = null,
public readonly string $healthStatus = 'unknown',
public readonly ?\DateTimeImmutable $healthLastTestedAt = null,
public readonly ?string $healthLastError = null,
) {
}
}

View File

@@ -43,6 +43,8 @@ class UpsertContentSourceResource
public readonly ?string $imageSelector = null,
public readonly ?string $nextPageSelector = null,
public readonly ?string $chapterSelector = null,
public readonly ?string $testSlug = null,
public readonly ?float $testChapterNumber = null,
) {
}
}

View File

@@ -0,0 +1,35 @@
<?php
namespace App\Domain\Setting\Infrastructure\ApiPlatform\State\Processor;
use ApiPlatform\Metadata\Operation;
use ApiPlatform\State\ProcessorInterface;
use App\Domain\Setting\Application\Command\DeleteContentSourceCommand;
use App\Domain\Setting\Application\CommandHandler\DeleteContentSourceCommandHandler;
use App\Domain\Setting\Domain\Exception\ContentSourceNotFoundException;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;
readonly class DeleteContentSourceStateProcessor implements ProcessorInterface
{
public function __construct(
private DeleteContentSourceCommandHandler $handler
) {
}
public function process(mixed $data, Operation $operation, array $uriVariables = [], array $context = []): int
{
if (!isset($uriVariables['id'])) {
throw new \InvalidArgumentException('Content source ID is required');
}
try {
$command = new DeleteContentSourceCommand((int) $uriVariables['id']);
$this->handler->handle($command);
return Response::HTTP_NO_CONTENT;
} catch (ContentSourceNotFoundException $e) {
throw new NotFoundHttpException($e->getMessage());
}
}
}

View File

@@ -30,6 +30,8 @@ readonly class UpsertContentSourceStateProcessor implements ProcessorInterface
imageSelector: $data->imageSelector,
nextPageSelector: $data->nextPageSelector,
chapterSelector: $data->chapterSelector,
testSlug: $data->testSlug,
testChapterNumber: $data->testChapterNumber,
);
$this->handler->handle($command);

View File

@@ -0,0 +1,39 @@
<?php
namespace App\Domain\Setting\Infrastructure\ApiPlatform\State\Provider;
use ApiPlatform\Metadata\Operation;
use ApiPlatform\State\ProviderInterface;
use App\Domain\Setting\Domain\Contract\Repository\ContentSourceRepositoryInterface;
use App\Domain\Setting\Domain\Exception\ContentSourceNotFoundException;
use App\Domain\Setting\Infrastructure\ApiPlatform\Resource\DeleteContentSourceResource;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;
readonly class DeleteContentSourceStateProvider implements ProviderInterface
{
public function __construct(
private ContentSourceRepositoryInterface $contentSourceRepository
) {
}
public function provide(Operation $operation, array $uriVariables = [], array $context = []): DeleteContentSourceResource
{
if (!isset($uriVariables['id'])) {
throw new NotFoundHttpException('Content source ID is required');
}
$id = (int) $uriVariables['id'];
try {
$contentSource = $this->contentSourceRepository->findById($id);
if (!$contentSource) {
throw new ContentSourceNotFoundException($id);
}
return new DeleteContentSourceResource($id);
} catch (ContentSourceNotFoundException $e) {
throw new NotFoundHttpException($e->getMessage());
}
}
}

View File

@@ -32,6 +32,11 @@ readonly class GetContentSourceStateProvider implements ProviderInterface
nextPageSelector: $response->nextPageSelector,
chapterSelector: $response->chapterSelector,
cleanBaseUrl: $response->cleanBaseUrl,
testSlug: $response->testSlug,
testChapterNumber: $response->testChapterNumber,
healthStatus: $response->healthStatus,
healthLastTestedAt: $response->healthLastTestedAt,
healthLastError: $response->healthLastError,
);
} catch (ContentSourceNotFoundException $e) {
throw new NotFoundHttpException($e->getMessage());

View File

@@ -30,6 +30,11 @@ readonly class ListContentSourceStateProvider implements ProviderInterface
nextPageSelector: $contentSourceResponse->nextPageSelector,
chapterSelector: $contentSourceResponse->chapterSelector,
cleanBaseUrl: $contentSourceResponse->cleanBaseUrl,
testSlug: $contentSourceResponse->testSlug,
testChapterNumber: $contentSourceResponse->testChapterNumber,
healthStatus: $contentSourceResponse->healthStatus,
healthLastTestedAt: $contentSourceResponse->healthLastTestedAt,
healthLastError: $contentSourceResponse->healthLastError,
),
$response->contentSources
);

View File

@@ -17,6 +17,11 @@ readonly class ContentSourceMapper
imageSelector: $entity->getImageSelector(),
nextPageSelector: $entity->getNextPageSelector(),
chapterSelector: $entity->getChapterSelector(),
testSlug: $entity->getTestSlug(),
testChapterNumber: $entity->getTestChapterNumber(),
healthStatus: $entity->getHealthStatus(),
healthLastTestedAt: $entity->getHealthLastTestedAt(),
healthLastError: $entity->getHealthLastError(),
);
}
@@ -29,7 +34,12 @@ readonly class ContentSourceMapper
->setScrapingType($contentSource->getScrapingType())
->setImageSelector($contentSource->getImageSelector())
->setNextPageSelector($contentSource->getNextPageSelector())
->setChapterSelector($contentSource->getChapterSelector());
->setChapterSelector($contentSource->getChapterSelector())
->setTestSlug($contentSource->getTestSlug())
->setTestChapterNumber($contentSource->getTestChapterNumber())
->setHealthStatus($contentSource->getHealthStatus())
->setHealthLastTestedAt($contentSource->getHealthLastTestedAt())
->setHealthLastError($contentSource->getHealthLastError());
return $entity;
}
@@ -41,7 +51,12 @@ readonly class ContentSourceMapper
->setScrapingType($contentSource->getScrapingType())
->setImageSelector($contentSource->getImageSelector())
->setNextPageSelector($contentSource->getNextPageSelector())
->setChapterSelector($contentSource->getChapterSelector());
->setChapterSelector($contentSource->getChapterSelector())
->setTestSlug($contentSource->getTestSlug())
->setTestChapterNumber($contentSource->getTestChapterNumber())
->setHealthStatus($contentSource->getHealthStatus())
->setHealthLastTestedAt($contentSource->getHealthLastTestedAt())
->setHealthLastError($contentSource->getHealthLastError());
return $entity;
}

View File

@@ -0,0 +1,102 @@
<?php
namespace App\Domain\Setting\Infrastructure\Persistence\Repository;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceForHealthCheckInterface;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceHealthRepositoryInterface;
use App\Domain\Scraping\Domain\Model\ValueObject\ContentSourceHealthCheckData;
use App\Entity\ContentSource as ContentSourceEntity;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\Mercure\HubInterface;
use Symfony\Component\Mercure\Update;
readonly class DoctrineContentSourceForHealthCheckRepository implements ContentSourceForHealthCheckInterface, ContentSourceHealthRepositoryInterface
{
public function __construct(
private EntityManagerInterface $entityManager,
private HubInterface $hub,
private LoggerInterface $logger,
) {
}
public function getAll(): array
{
$entities = $this->entityManager->getRepository(ContentSourceEntity::class)->findAll();
return array_map(
fn (ContentSourceEntity $entity) => new ContentSourceHealthCheckData(
id: $entity->getId(),
baseUrl: $entity->getBaseUrl(),
chapterUrlFormat: $entity->getChapterUrlFormat(),
scrapingType: $entity->getScrapingType(),
imageSelector: $entity->getImageSelector(),
nextPageSelector: $entity->getNextPageSelector(),
chapterSelector: $entity->getChapterSelector(),
testSlug: $entity->getTestSlug(),
testChapterNumber: $entity->getTestChapterNumber(),
),
$entities
);
}
public function markAsTesting(int $sourceId): void
{
$entity = $this->entityManager->find(ContentSourceEntity::class, $sourceId);
if (!$entity) {
return;
}
$entity->setHealthStatus('testing')
->setHealthLastError(null);
$this->entityManager->flush();
$this->publishUpdate($sourceId, 'testing', null);
}
public function markAsHealthy(int $sourceId, \DateTimeImmutable $testedAt): void
{
$entity = $this->entityManager->find(ContentSourceEntity::class, $sourceId);
if (!$entity) {
return;
}
$entity->setHealthStatus('ok')
->setHealthLastTestedAt($testedAt)
->setHealthLastError(null);
$this->entityManager->flush();
$this->publishUpdate($sourceId, 'ok', null);
}
public function markAsUnhealthy(int $sourceId, \DateTimeImmutable $testedAt, string $error): void
{
$entity = $this->entityManager->find(ContentSourceEntity::class, $sourceId);
if (!$entity) {
return;
}
$entity->setHealthStatus('ko')
->setHealthLastTestedAt($testedAt)
->setHealthLastError($error);
$this->entityManager->flush();
$this->publishUpdate($sourceId, 'ko', $error);
}
private function publishUpdate(int $sourceId, string $status, ?string $error): void
{
try {
$this->hub->publish(new Update(
"scrapers/health/{$sourceId}",
json_encode(['sourceId' => $sourceId, 'status' => $status, 'error' => $error])
));
} catch (\Throwable $e) {
$this->logger->warning('Mercure publish failed for scraper health update', [
'sourceId' => $sourceId,
'status' => $status,
'error' => $e->getMessage(),
]);
}
}
}

View File

@@ -36,6 +36,21 @@ class ContentSource
#[ORM\Column(length: 255, nullable: true)]
private ?string $ChapterSelector = null;
#[ORM\Column(length: 255, nullable: true)]
private ?string $testSlug = null;
#[ORM\Column(nullable: true)]
private ?float $testChapterNumber = null;
#[ORM\Column(length: 20, options: ['default' => 'unknown'])]
private string $healthStatus = 'unknown';
#[ORM\Column(nullable: true)]
private ?\DateTimeImmutable $healthLastTestedAt = null;
#[ORM\Column(type: 'text', nullable: true)]
private ?string $healthLastError = null;
public function getId(): ?int
{
return $this->id;
@@ -119,6 +134,66 @@ class ContentSource
return $this;
}
public function getTestSlug(): ?string
{
return $this->testSlug;
}
public function setTestSlug(?string $testSlug): static
{
$this->testSlug = $testSlug;
return $this;
}
public function getTestChapterNumber(): ?float
{
return $this->testChapterNumber;
}
public function setTestChapterNumber(?float $testChapterNumber): static
{
$this->testChapterNumber = $testChapterNumber;
return $this;
}
public function getHealthStatus(): string
{
return $this->healthStatus;
}
public function setHealthStatus(string $healthStatus): static
{
$this->healthStatus = $healthStatus;
return $this;
}
public function getHealthLastTestedAt(): ?\DateTimeImmutable
{
return $this->healthLastTestedAt;
}
public function setHealthLastTestedAt(?\DateTimeImmutable $healthLastTestedAt): static
{
$this->healthLastTestedAt = $healthLastTestedAt;
return $this;
}
public function getHealthLastError(): ?string
{
return $this->healthLastError;
}
public function setHealthLastError(?string $healthLastError): static
{
$this->healthLastError = $healthLastError;
return $this;
}
public function getCleanBaseUrl(): string
{
return preg_replace(

View File

@@ -3,13 +3,12 @@
namespace App\Scheduler;
use App\Message\RefreshAndDownloadChapters;
use Symfony\Component\Scheduler\Attribute\AsSchedule;
use Symfony\Component\Scheduler\RecurringMessage;
use Symfony\Component\Scheduler\Schedule;
use Symfony\Component\Scheduler\ScheduleProviderInterface;
use Symfony\Contracts\Cache\CacheInterface;
#[AsSchedule]
// Désactivé : remplacé par MonitoringSchedule (DDD) dans src/Domain/Manga/Infrastructure/Scheduler/
class MainSchedule implements ScheduleProviderInterface
{
public function __construct(private CacheInterface $cache)

View File

@@ -0,0 +1,27 @@
<?php
namespace App\Tests\Domain\Scraping\Adapter;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceForHealthCheckInterface;
use App\Domain\Scraping\Domain\Model\ValueObject\ContentSourceHealthCheckData;
class InMemoryContentSourceForHealthCheckRepository implements ContentSourceForHealthCheckInterface
{
/** @var ContentSourceHealthCheckData[] */
private array $sources = [];
public function add(ContentSourceHealthCheckData $data): void
{
$this->sources[] = $data;
}
public function getAll(): array
{
return $this->sources;
}
public function clear(): void
{
$this->sources = [];
}
}

View File

@@ -0,0 +1,41 @@
<?php
namespace App\Tests\Domain\Scraping\Adapter;
use App\Domain\Scraping\Domain\Contract\Repository\ContentSourceHealthRepositoryInterface;
class InMemoryContentSourceHealthRepository implements ContentSourceHealthRepositoryInterface
{
/** @var array<int, array{status: string, testedAt: ?\DateTimeImmutable, error: ?string}> */
private array $statuses = [];
public function markAsTesting(int $sourceId): void
{
$this->statuses[$sourceId] = ['status' => 'testing', 'testedAt' => null, 'error' => null];
}
public function markAsHealthy(int $sourceId, \DateTimeImmutable $testedAt): void
{
$this->statuses[$sourceId] = ['status' => 'ok', 'testedAt' => $testedAt, 'error' => null];
}
public function markAsUnhealthy(int $sourceId, \DateTimeImmutable $testedAt, string $error): void
{
$this->statuses[$sourceId] = ['status' => 'ko', 'testedAt' => $testedAt, 'error' => $error];
}
public function getStatus(int $sourceId): ?string
{
return $this->statuses[$sourceId]['status'] ?? null;
}
public function getError(int $sourceId): ?string
{
return $this->statuses[$sourceId]['error'] ?? null;
}
public function clear(): void
{
$this->statuses = [];
}
}

View File

@@ -0,0 +1,154 @@
<?php
namespace App\Tests\Domain\Scraping\Application\CommandHandler;
use App\Domain\Scraping\Application\Command\CheckAllScrapersHealth;
use App\Domain\Scraping\Application\CommandHandler\CheckAllScrapersHealthHandler;
use App\Domain\Scraping\Application\CommandHandler\TestScraperConfigurationHandler;
use App\Domain\Scraping\Domain\Model\ValueObject\ContentSourceHealthCheckData;
use App\Tests\Domain\Scraping\Adapter\InMemoryContentSourceForHealthCheckRepository;
use App\Tests\Domain\Scraping\Adapter\InMemoryContentSourceHealthRepository;
use App\Tests\Domain\Scraping\Adapter\InMemoryScraperAdapter;
use App\Tests\Domain\Scraping\Adapter\InMemoryScraperFactory;
use PHPUnit\Framework\TestCase;
use Psr\Log\NullLogger;
class CheckAllScrapersHealthHandlerTest extends TestCase
{
private InMemoryContentSourceForHealthCheckRepository $sourceRepo;
private InMemoryContentSourceHealthRepository $healthRepo;
private InMemoryScraperFactory $scraperFactory;
private CheckAllScrapersHealthHandler $handler;
protected function setUp(): void
{
$this->sourceRepo = new InMemoryContentSourceForHealthCheckRepository();
$this->healthRepo = new InMemoryContentSourceHealthRepository();
$this->scraperFactory = new InMemoryScraperFactory();
$this->scraperFactory->addScraper('html', new InMemoryScraperAdapter());
$testScraperHandler = new TestScraperConfigurationHandler($this->scraperFactory);
$this->handler = new CheckAllScrapersHealthHandler(
$this->sourceRepo,
$this->healthRepo,
$testScraperHandler,
new NullLogger(),
);
}
public function testSourceWithoutTestSlugIsSkipped(): void
{
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 1,
baseUrl: 'https://example.com',
chapterUrlFormat: 'https://example.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: null,
testChapterNumber: null,
));
$this->handler->handle(new CheckAllScrapersHealth());
$this->assertNull($this->healthRepo->getStatus(1));
}
public function testSourceWithTestSlugIsMarkedAsHealthyOnSuccess(): void
{
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 2,
baseUrl: 'https://example.com',
chapterUrlFormat: 'https://example.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: 'one-piece',
testChapterNumber: 1.0,
));
$this->handler->handle(new CheckAllScrapersHealth());
$this->assertSame('ok', $this->healthRepo->getStatus(2));
$this->assertNull($this->healthRepo->getError(2));
}
public function testSourceIsMarkedAsUnhealthyWhenScraperThrows(): void
{
$failingScraper = new InMemoryScraperAdapter();
$failingScraper->simulateError(new \RuntimeException('Connexion refusée'));
$this->scraperFactory->addScraper('html', $failingScraper);
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 3,
baseUrl: 'https://example.com',
chapterUrlFormat: 'https://example.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: 'one-piece',
testChapterNumber: 1.0,
));
$this->handler->handle(new CheckAllScrapersHealth());
$this->assertSame('ko', $this->healthRepo->getStatus(3));
$this->assertNotNull($this->healthRepo->getError(3));
}
public function testMultipleSourcesAreAllProcessed(): void
{
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 10,
baseUrl: 'https://siteA.com',
chapterUrlFormat: 'https://siteA.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: 'manga-a',
testChapterNumber: 1.0,
));
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 11,
baseUrl: 'https://siteB.com',
chapterUrlFormat: 'https://siteB.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: null,
testChapterNumber: null,
));
$this->sourceRepo->add(new ContentSourceHealthCheckData(
id: 12,
baseUrl: 'https://siteC.com',
chapterUrlFormat: 'https://siteC.com/{slug}/{chapterNumber}',
scrapingType: 'html',
imageSelector: 'img',
nextPageSelector: null,
chapterSelector: null,
testSlug: 'manga-c',
testChapterNumber: 3.0,
));
$this->handler->handle(new CheckAllScrapersHealth());
$this->assertSame('ok', $this->healthRepo->getStatus(10));
$this->assertNull($this->healthRepo->getStatus(11)); // skippée
$this->assertSame('ok', $this->healthRepo->getStatus(12));
}
protected function tearDown(): void
{
$this->sourceRepo->clear();
$this->healthRepo->clear();
$this->scraperFactory->clear();
}
}

View File

@@ -7,6 +7,7 @@ use App\Domain\Scraping\Application\CommandHandler\ScrapeChapterHandler;
use App\Domain\Scraping\Domain\Event\ChapterScrapingFailed;
use App\Domain\Scraping\Domain\Event\ChapterScrapingStarted;
use App\Domain\Scraping\Domain\Model\Chapter;
use App\Domain\Scraping\Domain\Model\ScrapingJob;
use App\Domain\Shared\Domain\Event\ChapterScraped;
use App\Tests\Domain\Scraping\Adapter\InMemoryChapterRepository;
use App\Tests\Domain\Scraping\Adapter\InMemoryEventBus;
@@ -16,8 +17,6 @@ use App\Tests\Domain\Scraping\Adapter\InMemoryMangaRepository;
use App\Tests\Domain\Scraping\Adapter\InMemoryScraperFactory;
use App\Tests\Domain\Scraping\Adapter\InMemorySourceRepository;
use App\Tests\Domain\Shared\Adapter\InMemoryJobRepository;
use Doctrine\ORM\EntityManagerInterface;
use PHPUnit\Framework\MockObject\MockObject;
use PHPUnit\Framework\TestCase;
class ScrapeChapterHandlerTest extends TestCase
@@ -30,7 +29,6 @@ class ScrapeChapterHandlerTest extends TestCase
private InMemoryMangaRepository $mangaRepository;
private InMemorySourceRepository $sourceRepository;
private InMemoryEventBus $eventBus;
private EntityManagerInterface|MockObject $entityManager;
private ScrapeChapterHandler $handler;
protected function setUp(): void
@@ -43,11 +41,6 @@ class ScrapeChapterHandlerTest extends TestCase
$this->mangaRepository = new InMemoryMangaRepository();
$this->sourceRepository = new InMemorySourceRepository();
$this->eventBus = new InMemoryEventBus();
$this->entityManager = $this->createMock(EntityManagerInterface::class);
$this->entityManager->method('beginTransaction')->willReturn(null);
$this->entityManager->method('commit')->willReturn(null);
$this->entityManager->method('rollback')->willReturn(null);
$this->chapterRepository->save(new Chapter(
id: '1',
@@ -65,26 +58,27 @@ class ScrapeChapterHandlerTest extends TestCase
$this->mangaRepository,
$this->sourceRepository,
$this->eventBus,
$this->entityManager
);
}
public function testHandleSuccessfully(): void
{
$command = new ScrapeChapter(
chapterId: '1'
);
$jobId = 'test-job-id';
$job = new ScrapingJob($jobId, 'test-manga', 2);
$this->jobRepository->save($job);
$command = new ScrapeChapter(chapterId: '1', jobId: $jobId);
$this->handler->handle($command);
$job = $this->jobRepository->findByType('scraping_job');
$this->assertCount(1, $job);
$job = array_values($job)[0];
$jobs = $this->jobRepository->findByType('scraping_job');
$this->assertCount(1, $jobs);
$job = array_values($jobs)[0];
$dispatchedMessages = $this->eventBus->getDispatchedMessages();
$this->assertCount(2, $dispatchedMessages);
$this->assertInstanceOf(ChapterScrapingStarted::class, $dispatchedMessages[0]);
$this->assertSame($jobId, $dispatchedMessages[0]->getJobId());
$this->assertSame(2.0, $dispatchedMessages[0]->getChapterNumber());
$this->assertInstanceOf(ChapterScraped::class, $dispatchedMessages[1]);

View File

@@ -0,0 +1,72 @@
<?php
declare(strict_types=1);
namespace App\Tests\Feature\Scraping;
use App\Entity\ContentSource;
use App\Tests\Feature\AbstractApiTestCase;
use Symfony\Component\HttpFoundation\Response;
use Zenstruck\Foundry\Test\ResetDatabase;
final class CheckAllScrapersHealthTest extends AbstractApiTestCase
{
use ResetDatabase;
private function post(): void
{
static::createClient()->request('POST', '/api/scraping/check-all-health', [
'json' => new \stdClass(),
]);
}
public function testItReturns202WithNoSources(): void
{
$this->post();
$this->assertResponseStatusCodeSame(Response::HTTP_ACCEPTED);
}
public function testItReturns202WithSourcesHavingNoTestConfig(): void
{
$source = new ContentSource();
$source->setBaseUrl('https://example.com')
->setChapterUrlFormat('https://example.com/{slug}/{chapterNumber}')
->setScrapingType('html');
$this->entityManager->persist($source);
$this->entityManager->flush();
$this->post();
$this->assertResponseStatusCodeSame(Response::HTTP_ACCEPTED);
// La source sans testSlug ne doit pas avoir son statut modifié
$this->entityManager->clear();
$reloaded = $this->entityManager->find(ContentSource::class, $source->getId());
$this->assertSame('unknown', $reloaded->getHealthStatus());
}
public function testHealthStatusIsUpdatedForSourcesWithTestConfig(): void
{
$source = new ContentSource();
$source->setBaseUrl('https://example.com')
->setChapterUrlFormat('https://example.com/{slug}/{chapterNumber}')
->setScrapingType('html')
->setTestSlug('one-piece')
->setTestChapterNumber(1.0);
$this->entityManager->persist($source);
$this->entityManager->flush();
$this->post();
$this->assertResponseStatusCodeSame(Response::HTTP_ACCEPTED);
// Le statut ne doit plus être 'unknown' après le test
$this->entityManager->clear();
$reloaded = $this->entityManager->find(ContentSource::class, $source->getId());
$this->assertNotSame('unknown', $reloaded->getHealthStatus());
$this->assertNotSame('testing', $reloaded->getHealthStatus()); // doit être terminé
}
}

View File

@@ -35,13 +35,14 @@ class ScrapeChapterTest extends AbstractApiTestCase
// Then
$this->assertResponseStatusCodeSame(202);
$messages = $this->messageBus->getDispatchedMessages();
$messages = InMemoryMessageBus::$messages;
$this->assertCount(1, $messages, 'Un message devrait être dispatché');
/** @var ScrapeChapter $message */
$message = $messages[0];
$this->assertInstanceOf(ScrapeChapter::class, $message);
$this->assertEquals('chapter-123', $message->chapterId);
$this->assertNotEmpty($message->jobId);
}
public function testInitiateChapterScrapingWithInvalidPayload(): void
@@ -72,6 +73,6 @@ class ScrapeChapterTest extends AbstractApiTestCase
protected function tearDown(): void
{
parent::tearDown();
$this->messageBus->clear();
InMemoryMessageBus::$messages = [];
}
}

View File

@@ -0,0 +1,50 @@
<?php
declare(strict_types=1);
namespace App\Tests\Feature\Setting;
use App\Entity\ContentSource;
use App\Tests\Feature\AbstractApiTestCase;
use Symfony\Component\HttpFoundation\Response;
use Zenstruck\Foundry\Test\ResetDatabase;
final class DeleteContentSourceTest extends AbstractApiTestCase
{
use ResetDatabase;
private int $sourceId;
protected function setUp(): void
{
parent::setUp();
$source = new ContentSource();
$source->setBaseUrl('https://mangadex.org')
->setChapterUrlFormat('https://mangadex.org/chapter/{id}')
->setScrapingType('html');
$this->entityManager->persist($source);
$this->entityManager->flush();
$this->sourceId = $source->getId();
}
public function testItDeletesSourceSuccessfully(): void
{
static::createClient()->request('DELETE', "/api/content-sources/{$this->sourceId}");
$this->assertResponseStatusCodeSame(Response::HTTP_NO_CONTENT);
$this->entityManager->clear();
$deletedSource = $this->entityManager->find(ContentSource::class, $this->sourceId);
$this->assertNull($deletedSource);
}
public function testItReturnsNotFoundWhenSourceDoesNotExist(): void
{
static::createClient()->request('DELETE', '/api/content-sources/999999');
$this->assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
}
}